-- Logs begin at Wed 2022-08-24 15:03:26 UTC, end at Wed 2022-08-24 16:37:15 UTC. -- Aug 24 15:32:12.565493 dv-cinder-849325-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Aug 24 15:32:12.574971 dv-cinder-849325-3-iscsi devstack@keystone.service[88206]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Aug 24 15:32:12.576924 dv-cinder-849325-3-iscsi devstack@keystone.service[88206]: *** Starting uWSGI 2.0.18-debian (64bit) on [Wed Aug 24 15:32:12 2022] *** Aug 24 15:32:12.576924 dv-cinder-849325-3-iscsi devstack@keystone.service[88206]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Aug 24 15:32:12.576924 dv-cinder-849325-3-iscsi devstack@keystone.service[88206]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Aug 24 15:32:12.576924 dv-cinder-849325-3-iscsi devstack@keystone.service[88206]: nodename: dv-cinder-849325-3-iscsi Aug 24 15:32:12.576924 dv-cinder-849325-3-iscsi devstack@keystone.service[88206]: machine: x86_64 Aug 24 15:32:12.576924 dv-cinder-849325-3-iscsi devstack@keystone.service[88206]: clock source: unix Aug 24 15:32:12.576924 dv-cinder-849325-3-iscsi devstack@keystone.service[88206]: pcre jit disabled Aug 24 15:32:12.576924 dv-cinder-849325-3-iscsi devstack@keystone.service[88206]: detected number of CPU cores: 4 Aug 24 15:32:12.576924 dv-cinder-849325-3-iscsi devstack@keystone.service[88206]: current working directory: / Aug 24 15:32:12.576924 dv-cinder-849325-3-iscsi devstack@keystone.service[88206]: detected binary path: /usr/bin/uwsgi-core Aug 24 15:32:12.576924 dv-cinder-849325-3-iscsi devstack@keystone.service[88206]: your processes number limit is 47680 Aug 24 15:32:12.576924 dv-cinder-849325-3-iscsi devstack@keystone.service[88206]: your memory page size is 4096 bytes Aug 24 15:32:12.576924 dv-cinder-849325-3-iscsi devstack@keystone.service[88206]: detected max file descriptor number: 2048 Aug 24 15:32:12.576924 dv-cinder-849325-3-iscsi devstack@keystone.service[88206]: lock engine: pthread robust mutexes Aug 24 15:32:12.577881 dv-cinder-849325-3-iscsi devstack@keystone.service[88206]: thunder lock: enabled Aug 24 15:32:12.577881 dv-cinder-849325-3-iscsi devstack@keystone.service[88206]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Aug 24 15:32:12.577881 dv-cinder-849325-3-iscsi devstack@keystone.service[88206]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Aug 24 15:32:12.624794 dv-cinder-849325-3-iscsi devstack@keystone.service[88206]: Python main interpreter initialized at 0x5646f70789c0 Aug 24 15:32:12.624794 dv-cinder-849325-3-iscsi devstack@keystone.service[88206]: python threads support enabled Aug 24 15:32:12.624794 dv-cinder-849325-3-iscsi devstack@keystone.service[88206]: your server socket listen backlog is limited to 100 connections Aug 24 15:32:12.624794 dv-cinder-849325-3-iscsi devstack@keystone.service[88206]: your mercy for graceful operations on workers is 90 seconds Aug 24 15:32:12.625179 dv-cinder-849325-3-iscsi devstack@keystone.service[88206]: mapped 403077 bytes (393 KB) for 2 cores Aug 24 15:32:12.625254 dv-cinder-849325-3-iscsi devstack@keystone.service[88206]: *** Operational MODE: preforking *** Aug 24 15:32:12.625517 dv-cinder-849325-3-iscsi devstack@keystone.service[88206]: *** uWSGI is running in multiple interpreter mode *** Aug 24 15:32:12.625517 dv-cinder-849325-3-iscsi devstack@keystone.service[88206]: spawned uWSGI master process (pid: 88206) Aug 24 15:32:12.625730 dv-cinder-849325-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Aug 24 15:32:12.626578 dv-cinder-849325-3-iscsi devstack@keystone.service[88206]: spawned uWSGI worker 1 (pid: 88217, cores: 1) Aug 24 15:32:12.626669 dv-cinder-849325-3-iscsi devstack@keystone.service[88206]: spawned uWSGI worker 2 (pid: 88218, cores: 1) Aug 24 15:32:12.626809 dv-cinder-849325-3-iscsi devstack@keystone.service[88206]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Aug 24 15:32:16.990383 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 24 15:32:16.990383 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: from cryptography.utils import int_from_bytes Aug 24 15:32:16.991179 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 24 15:32:16.991179 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: from cryptography.utils import int_from_bytes Aug 24 15:32:16.998939 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 24 15:32:16.998939 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: from cryptography.utils import int_from_bytes Aug 24 15:32:16.999484 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 24 15:32:16.999484 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: from cryptography.utils import int_from_bytes Aug 24 15:32:17.042302 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Aug 24 15:32:17.042510 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Aug 24 15:32:17.042667 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] command line args: [] {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Aug 24 15:32:17.042933 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Aug 24 15:32:17.043113 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ================================================================================ {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Aug 24 15:32:17.043363 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] admin_token = **** {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.043600 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] config_dir = [] {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.043812 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.043991 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] config_source = [] {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.044171 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] debug = True {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.044370 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.044754 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.044933 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] insecure_debug = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.045185 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.045436 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.045665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] list_limit = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.045839 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] log_config_append = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.046015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.046190 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] log_dir = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.046372 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] log_file = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.046592 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.046839 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.047081 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.047320 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.047497 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.047697 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.047941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.048186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.048416 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.048644 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.048872 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.049099 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.049343 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.049557 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Aug 24 15:32:17.049675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] notification_format = cadf {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.049793 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Aug 24 15:32:17.049905 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.050025 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] command line args: [] {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Aug 24 15:32:17.050203 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Aug 24 15:32:17.050316 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.102/identity {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.050429 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ================================================================================ {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Aug 24 15:32:17.050549 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] publish_errors = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.050668 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] admin_token = **** {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.050779 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.050897 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] config_dir = [] {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.051007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.051130 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.051250 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.051362 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] config_source = [] {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.051478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.051590 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] debug = True {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.051729 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.051853 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.051978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] standard_threads = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.052080 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.052180 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] strict_password_check = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.052274 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] insecure_debug = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.052380 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.052485 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.052585 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] use_eventlog = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.052703 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.052816 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] use_journal = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.052933 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] list_limit = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.052933 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] log_config_append = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.053163 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] use_json = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.053319 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.053441 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] use_stderr = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.053554 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] log_dir = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.053665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] use_syslog = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.053776 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] log_file = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.053896 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] watch_log_file = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.054008 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.054119 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.054230 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.054230 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.054472 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.054595 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.054820 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.054958 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.055075 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.055186 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.055186 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.055412 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.055523 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.055656 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.055809 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.056055 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.056191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.056306 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.056419 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.056536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.056713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.056837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.056955 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.057067 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.057178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.057294 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] notification_format = cadf {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.057414 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.057518 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.057613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.057819 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.058011 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.102/identity {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.058136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.058249 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] publish_errors = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.058360 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.058471 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.058589 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.058694 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.058809 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.058912 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.059022 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.059134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.059251 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.059375 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.059493 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.059610 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] standard_threads = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.059761 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.059874 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] strict_password_check = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.059990 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.060110 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.060222 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] use_eventlog = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.060331 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.060447 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] use_journal = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.060559 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.060670 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] use_json = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.060788 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.060901 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] use_stderr = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.060901 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] use_syslog = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.061109 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.061219 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] watch_log_file = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 15:32:17.061329 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.061447 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.061566 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.061687 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.061804 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.061925 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.062042 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.062042 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.062330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.062441 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.062557 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.062668 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.062778 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.062896 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.063009 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.063119 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.063229 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.063339 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.063455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.063574 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.063726 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.063822 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.063914 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.064007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.064099 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.064205 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.064317 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.064428 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.064544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.064661 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.064782 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.064939 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.065060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.065173 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.065291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.065410 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.065531 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.065644 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.065644 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.065859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.065978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.066091 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.066211 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.066324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.066434 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.066545 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.066656 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.066772 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.066893 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.067006 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.067124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.067242 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.067354 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.067465 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.067668 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.067793 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.067906 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.068018 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.068129 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.068240 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.068360 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.068729 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.068845 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.068956 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.069067 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.069178 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.069297 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.069412 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.069515 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.069620 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.069715 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.069808 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.069907 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.069907 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.070080 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.070172 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.070264 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.070264 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.070525 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.070622 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.070713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.070989 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.071274 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.071432 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.071562 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.071734 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.071851 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.071955 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.072077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.072200 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.072323 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] cache.enabled = True {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.072437 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.072541 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.072657 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.072770 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.072883 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.072992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.073113 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.073229 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.073350 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.073471 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.073575 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.073691 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.073795 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.073908 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.074020 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.074133 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.074247 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.074359 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.074469 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.074582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.074695 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.074803 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.074916 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.075070 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.075196 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.075318 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.075432 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.075545 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.075671 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.075785 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.075915 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.076031 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.076144 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.076257 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.076371 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.076483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.076605 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.076715 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.076811 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.076922 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.077044 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.077159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.077280 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.077396 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.077509 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.077622 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.077734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.077846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.077967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.078088 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.078202 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.078320 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.078436 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.078550 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.078671 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.078786 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.078899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] auth.external = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.078996 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.079107 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] auth.mapped = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.079220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.079406 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.079527 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.079685 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.079801 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.079928 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] auth.password = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.080044 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] cache.enabled = True {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.080170 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] auth.token = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.080293 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.080400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.080497 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.080591 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] catalog.caching = True {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.080703 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.080816 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.080937 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.081052 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.081165 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.081277 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.081396 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.081516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.081640 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.081755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.081868 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.081980 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] credential.caching = True {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.082093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.082206 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] credential.driver = sql {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.082327 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.082442 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.082555 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.082669 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.082782 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.082895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.082895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.083129 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.083244 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.083385 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.083510 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.083640 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.083640 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.083867 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.083995 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.084111 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.084225 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.084338 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.084458 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.084572 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.084701 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.084819 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.084933 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.085080 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.085197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.085311 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.085436 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.085558 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.085679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.085796 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] federation.caching = True {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.086457 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.086579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.086801 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.086928 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] federation.driver = sql {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.087046 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.087160 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.087160 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.087383 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.087506 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.087622 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.087766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] auth.external = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.087881 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.087995 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] auth.mapped = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.088108 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.088233 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.088351 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.088471 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.088586 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.088700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] auth.password = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.088813 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.088936 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] auth.token = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.089058 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] identity.caching = True {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.089179 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.089300 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.089415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] catalog.caching = True {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.089529 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.089655 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.089770 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.089869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.089982 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.090095 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.090209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] identity.driver = sql {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.090330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.090450 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.090564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.090678 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.090792 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] credential.caching = True {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.090905 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.091031 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] credential.driver = sql {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.091146 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.091259 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.091373 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.091485 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.091598 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.091742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.091859 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.091859 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.092073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.092073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.092275 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.092275 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.092453 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.092547 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.092649 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.092761 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.092969 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.093090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.093204 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.093318 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.093431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.093544 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.093731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.093849 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.093963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.094082 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.094199 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.094321 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.094444 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.094561 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.094683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] federation.caching = True {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.094839 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.094968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.095123 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.095252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] federation.driver = sql {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.095368 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.095483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.095598 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.095764 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.095879 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.096006 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.096123 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.096238 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.096351 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.096465 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.096579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.096701 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.096817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.096817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.097029 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.097143 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.097256 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.097380 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.password = **** {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.097496 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.097609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.097723 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] identity.caching = True {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.097842 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.097957 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.097957 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.098181 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.098295 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.098409 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.098522 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.098635 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.098756 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] identity.driver = sql {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.098872 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.098986 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.099100 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.099197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.099309 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.099415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.099529 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.099661 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.099778 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.099891 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.099988 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.100113 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.100229 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.100344 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.100463 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.100577 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.100690 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.100690 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.user = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.100916 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.101030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.101144 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.101258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.101371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.101493 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.101720 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.102364 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.102463 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.102585 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.102708 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.102836 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.102836 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.102836 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.103151 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.103248 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.103371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.103488 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.103603 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.103770 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.103886 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.104000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.104126 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.104243 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.104358 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.104471 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.104585 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.104700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.104822 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.104938 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.105052 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.105203 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.105321 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.105435 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.105557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.105674 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.105788 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.105901 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.105998 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.106111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.106233 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.106349 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.106463 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.106576 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.106689 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.106803 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.106925 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.107040 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.password = **** {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.107153 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.107266 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.107379 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] policy.driver = sql {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.107493 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.107613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.107749 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.107863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.107976 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.108090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.108203 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.108328 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] receipt.caching = True {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.108444 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.108557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.108670 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.108784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.108898 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.109025 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.109140 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.109253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.109367 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.109480 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.109593 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.109718 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] resource.caching = True {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.109834 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.109949 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.110062 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.110176 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] resource.driver = sql {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.110289 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.110411 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.110530 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.user = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.110644 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.110757 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.110870 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.110996 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.111120 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] revoke.caching = True {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.111235 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.111349 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.111462 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.111581 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.111727 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.111854 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] role.cache_time = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.111972 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.112086 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] role.caching = True {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.112199 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.112313 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] role.driver = sql {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.112426 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.112548 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] role.list_limit = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.112663 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.112777 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.112891 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.113004 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.113120 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.113242 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.113358 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.113478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.113478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.113694 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.113807 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.113929 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.114044 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.114166 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.114281 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.114395 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.114509 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.114631 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.114746 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.114860 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.114973 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.115086 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.115201 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.115356 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.115475 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.115589 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.115589 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.115821 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.115947 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.116063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.116177 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.116291 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.116404 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.116517 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.116705 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.116823 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.116937 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] policy.driver = sql {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.117050 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.117208 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.117329 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.117452 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.117568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.117568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.117781 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.117896 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.117896 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.118105 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.118219 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] receipt.caching = True {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.118334 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.118447 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.118561 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.118674 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.118796 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.118911 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.119024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.119137 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.119250 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.119364 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.119485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.119612 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] resource.caching = True {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.119778 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] token.cache_time = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.119893 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.120006 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] token.caching = True {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.120120 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] resource.driver = sql {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.120246 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.120363 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.120481 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] token.provider = fernet {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.120595 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.120709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.120822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.120957 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.121077 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] revoke.caching = True {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.121191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.121305 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.121419 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.121533 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.121655 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.121770 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] role.cache_time = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.121884 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.121997 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] role.caching = True {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.122111 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] trust.driver = sql {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.122236 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] role.driver = sql {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.122361 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.122478 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] role.list_limit = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.122591 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.122758 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.122878 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.122878 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.123102 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.123219 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.123333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.123454 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.123570 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.123702 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.123830 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.123946 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.124060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.124180 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] database.connection = **** {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.124303 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.124427 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.124556 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.124680 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.124796 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.124910 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.125023 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.125136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.125250 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.125408 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.125528 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.125643 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.125756 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.125869 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.125869 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.126091 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.126208 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.126322 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.126436 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.126549 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.126665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.126787 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.126903 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.127017 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.127131 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.127245 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.127359 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.127481 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.127596 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.127729 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.127844 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.127957 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.128071 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.128197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.128392 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.128513 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.128627 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=88217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Aug 24 15:32:17.128749 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.128749 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.128962 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.129188 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.129502 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.129741 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.129903 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.130136 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] token.cache_time = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.130375 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] token.caching = True {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.130616 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.130875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] token.provider = fernet {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.131114 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.131375 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.131693 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.131889 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.132138 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.132385 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.132637 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] trust.driver = sql {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.132910 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.133684 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.133932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.134189 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.134447 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.134683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.134934 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.135232 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.135500 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] database.connection = **** {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.135786 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.136046 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.136290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.136537 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.136779 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.137023 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.137263 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.137505 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.137754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.137995 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.138243 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.138483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.138736 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.138976 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.139224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.139457 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.139724 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.139963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 15:32:17.140136 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=88218) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Aug 24 15:32:17.158669 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=88217) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.158890 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=88217) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.159110 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=88217) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.168183 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=88218) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.168328 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=88217) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.168409 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=88218) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.168635 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=88218) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.177226 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=88218) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.197206 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=88217) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.197472 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=88217) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.199112 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=88218) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.199349 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=88218) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.251597 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=88217) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.253064 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=88218) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.301758 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=88217) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.301983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=88217) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.304217 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=88218) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.304459 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=88218) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.461513 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=88217) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.461757 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=88217) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.461969 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=88217) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.462190 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=88217) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.462275 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=88218) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.462405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=88217) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.462489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=88218) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.462618 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=88217) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.462703 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=88218) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.462902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=88218) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.463122 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=88218) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.463341 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=88218) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.485478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=88217) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.485703 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=88217) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.485827 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=88218) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.485957 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=88217) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.486081 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=88218) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.486227 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=88217) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.486344 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=88218) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.486472 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=88217) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.486593 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=88218) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.486711 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=88217) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.486827 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=88218) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.486960 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=88217) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.487073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=88218) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.487073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=88218) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.497997 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=88217) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.498966 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=88218) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.511401 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=88217) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.511646 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=88217) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.511869 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=88217) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.511981 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=88218) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.512090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=88217) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.512219 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=88218) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.512292 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=88217) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.512450 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=88218) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.512519 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=88217) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.512688 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=88218) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.512780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=88217) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.512897 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=88218) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.513121 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=88218) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.513336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=88218) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 15:32:17.530869 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.531074 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 24 15:32:17.531184 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.531398 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 24 15:32:17.531478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.531606 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 24 15:32:17.531845 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.531928 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.532052 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 24 15:32:17.532140 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.532334 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.532603 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.532681 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.532868 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.532945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.533132 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.542144 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.542416 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.542687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.542973 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.543232 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.543517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.543820 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.544115 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.549877 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=88217) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.551018 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=88218) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.552759 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=88217) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.553039 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.553317 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.553600 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.553878 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.553960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=88218) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.554136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.554218 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.554405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.554496 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.554876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.555239 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.555364 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.555539 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.555654 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.555838 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.555921 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.556117 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.556398 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.556711 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.556960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.557234 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.579151 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.580646 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=88217) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.580927 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.581057 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.582530 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=88218) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.582817 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.585551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=88217) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.585948 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.586742 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.588656 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=88218) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.588951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.589238 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.593050 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=88217) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.593340 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.595555 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=88218) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.595871 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.597385 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=88217) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.597679 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.597962 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.598245 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.598521 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.598794 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.599070 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.599335 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.599937 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=88218) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.600234 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.600528 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.600816 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.601103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.601386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.602213 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.602501 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.616348 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.616628 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.620897 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.621186 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.621400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=88217) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.624609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.624905 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.625026 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=88218) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.628286 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.628592 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.629126 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=88217) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.632175 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=88217) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.633438 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=88218) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.635942 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.636239 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.636522 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.636814 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.636957 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=88218) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.637091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.637384 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.640197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.640497 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.640790 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.641080 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.641364 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.641661 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.653520 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=88217) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.653815 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.654136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.654345 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.658034 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=88218) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.658338 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.658605 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.658882 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.661148 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.662650 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.662920 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.665625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=88217) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.665907 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.666199 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.666337 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.667209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.667209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.667906 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.668190 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.670806 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=88218) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.671107 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.671405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.671721 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.672017 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.678196 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=88217) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.678485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.678768 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.679047 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.679327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.680203 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.680524 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.684483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=88218) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.684777 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.685066 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.685352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.685636 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.685938 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.686217 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.697117 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=88217) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.701742 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=88217) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.703342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=88218) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.705060 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.706405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=88218) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.706521 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.707978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=88217) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.708263 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.708562 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.709411 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.710959 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.713082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=88218) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.713312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.713601 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.715271 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.716751 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=88217) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.719660 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.719948 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.720099 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.720232 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.720516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.721573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=88218) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.724541 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.724835 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.725131 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.725419 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.728642 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=88217) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.728935 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.729216 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.733945 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=88218) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.734247 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.734530 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.736061 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=88217) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.736391 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.736626 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.736902 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.737224 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.737559 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.737743 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.738010 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.738288 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.738573 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.738856 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.739157 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.739395 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.739765 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=88217) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.741410 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=88218) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 15:32:17.741695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.742526 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.742969 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.743293 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.743607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.743904 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.744184 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.744530 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.744752 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.745038 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.745317 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.745599 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.745865 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=88218) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 15:32:17.774785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x5646f70789c0 pid: 88217 (default app) Aug 24 15:32:17.778901 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca216f9-2e6e-46de-8a35-d80dcd5a3a1a None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:17.779188 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca216f9-2e6e-46de-8a35-d80dcd5a3a1a None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:17.779459 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca216f9-2e6e-46de-8a35-d80dcd5a3a1a None None] PATH_INFO: `/v3` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:17.781527 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x5646f70789c0 pid: 88218 (default app) Aug 24 15:32:17.783573 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1/1] 10.222.0.102 () {50 vars in 796 bytes} [Wed Aug 24 15:32:17 2022] GET /identity/v3/ => generated 256 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:24.653240 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625245d8-0bc0-4405-aff1-7b7a2352ca1c None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:24.653860 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625245d8-0bc0-4405-aff1-7b7a2352ca1c None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:24.653860 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625245d8-0bc0-4405-aff1-7b7a2352ca1c None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:24.658324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1/2] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:32:24 2022] GET /identity => generated 271 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:32:24.666276 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fbddd1-1fec-46ad-902c-5d58fa680f02 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:24.666783 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fbddd1-1fec-46ad-902c-5d58fa680f02 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:24.666874 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fbddd1-1fec-46ad-902c-5d58fa680f02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:24.739160 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: INFO dbcounter [None req-f9fbddd1-1fec-46ad-902c-5d58fa680f02 None None] Registered counter for database keystone Aug 24 15:32:24.759742 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG oslo_db.sqlalchemy.engines [None req-f9fbddd1-1fec-46ad-902c-5d58fa680f02 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=88217) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Aug 24 15:32:24.767073 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writer thread running {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Aug 24 15:32:24.957935 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG passlib.handlers.bcrypt [None req-f9fbddd1-1fec-46ad-902c-5d58fa680f02 None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=88217) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Aug 24 15:32:24.957935 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG passlib.handlers.bcrypt [None req-f9fbddd1-1fec-46ad-902c-5d58fa680f02 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=88217) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Aug 24 15:32:25.016489 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-f9fbddd1-1fec-46ad-902c-5d58fa680f02 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:32:25.018121 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-f9fbddd1-1fec-46ad-902c-5d58fa680f02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:32:25.152452 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 2/3] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:32:24 2022] POST /identity/v3/auth/tokens => generated 915 bytes in 487 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Aug 24 15:32:25.634007 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625245d8-0bc0-4405-aff1-7b7a2352ca1c None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:25.706280 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: INFO dbcounter [None req-625245d8-0bc0-4405-aff1-7b7a2352ca1c None None] Registered counter for database keystone Aug 24 15:32:25.729722 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG oslo_db.sqlalchemy.engines [None req-625245d8-0bc0-4405-aff1-7b7a2352ca1c None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=88218) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Aug 24 15:32:25.736473 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writer thread running {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Aug 24 15:32:25.864848 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625245d8-0bc0-4405-aff1-7b7a2352ca1c None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:25.869778 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99de6980-398f-445d-9402-052a762538d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:25.870402 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99de6980-398f-445d-9402-052a762538d2 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:25.870687 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99de6980-398f-445d-9402-052a762538d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:25.870961 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99de6980-398f-445d-9402-052a762538d2 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:25.879394 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99de6980-398f-445d-9402-052a762538d2 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:25.932032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: WARNING keystone.common.rbac_enforcer.enforcer [None req-99de6980-398f-445d-9402-052a762538d2 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Aug 24 15:32:25.932296 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99de6980-398f-445d-9402-052a762538d2 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:25.942807 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 2/4] 10.222.0.102 () {58 vars in 1329 bytes} [Wed Aug 24 15:32:25 2022] GET /identity/v3/auth/tokens => generated 915 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 24 15:32:25.949366 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9fbddd1-1fec-46ad-902c-5d58fa680f02 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:25.964026 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9fbddd1-1fec-46ad-902c-5d58fa680f02 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:25.968610 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7beec395-17c6-4abf-9a1f-6b4a6b69c8fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:25.969763 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7beec395-17c6-4abf-9a1f-6b4a6b69c8fe admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:25.970053 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7beec395-17c6-4abf-9a1f-6b4a6b69c8fe admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:25.970333 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7beec395-17c6-4abf-9a1f-6b4a6b69c8fe admin admin] PATH_INFO: `/v3/projects/501a315437844349bdc02dafe337b855` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:25.971998 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7beec395-17c6-4abf-9a1f-6b4a6b69c8fe admin admin] RBAC: Authorizing `identity:get_project(project_id=501a315437844349bdc02dafe337b855)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:26.025862 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: WARNING keystone.common.rbac_enforcer.enforcer [None req-7beec395-17c6-4abf-9a1f-6b4a6b69c8fe admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Aug 24 15:32:26.026203 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7beec395-17c6-4abf-9a1f-6b4a6b69c8fe admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:26.030809 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 3/5] 10.222.0.102 () {56 vars in 1212 bytes} [Wed Aug 24 15:32:25 2022] GET /identity/v3/projects/501a315437844349bdc02dafe337b855 => generated 345 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:28.022549 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eae9bb0-2b83-42d5-9686-c3d0e01b62f4 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:28.023103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eae9bb0-2b83-42d5-9686-c3d0e01b62f4 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:28.023103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eae9bb0-2b83-42d5-9686-c3d0e01b62f4 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:28.027398 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 3/6] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:32:28 2022] GET /identity => generated 271 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:32:28.035930 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3445f501-1625-4c24-860c-4aad6b40f843 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:28.036505 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3445f501-1625-4c24-860c-4aad6b40f843 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:28.036505 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3445f501-1625-4c24-860c-4aad6b40f843 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:28.058579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-3445f501-1625-4c24-860c-4aad6b40f843 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:32:28.060318 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-3445f501-1625-4c24-860c-4aad6b40f843 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:32:28.070541 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 4/7] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:32:28 2022] POST /identity/v3/auth/tokens => generated 915 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Aug 24 15:32:28.344990 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eae9bb0-2b83-42d5-9686-c3d0e01b62f4 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:28.363612 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eae9bb0-2b83-42d5-9686-c3d0e01b62f4 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:28.368530 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0ea377-4e58-4375-b39c-add6a8c852c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:28.369139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0ea377-4e58-4375-b39c-add6a8c852c4 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:28.369415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0ea377-4e58-4375-b39c-add6a8c852c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:28.369684 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0ea377-4e58-4375-b39c-add6a8c852c4 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:28.378877 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f0ea377-4e58-4375-b39c-add6a8c852c4 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:28.381651 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f0ea377-4e58-4375-b39c-add6a8c852c4 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:28.398983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 4/8] 10.222.0.102 () {58 vars in 1329 bytes} [Wed Aug 24 15:32:28 2022] GET /identity/v3/auth/tokens => generated 915 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 24 15:32:28.405053 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3445f501-1625-4c24-860c-4aad6b40f843 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:28.415671 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3445f501-1625-4c24-860c-4aad6b40f843 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:28.420214 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200de367-0229-41e7-9e57-c9f3adef0b14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:28.420850 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200de367-0229-41e7-9e57-c9f3adef0b14 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:28.421131 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200de367-0229-41e7-9e57-c9f3adef0b14 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:28.421399 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200de367-0229-41e7-9e57-c9f3adef0b14 admin admin] PATH_INFO: `/v3/users/3eb7904f17244b85baf19c438c53917f` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:28.422725 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200de367-0229-41e7-9e57-c9f3adef0b14 admin admin] RBAC: Authorizing `identity:get_user(user_id=3eb7904f17244b85baf19c438c53917f)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:28.424606 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200de367-0229-41e7-9e57-c9f3adef0b14 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:28.426181 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 5/9] 10.222.0.102 () {56 vars in 1203 bytes} [Wed Aug 24 15:32:28 2022] GET /identity/v3/users/3eb7904f17244b85baf19c438c53917f => generated 248 bytes in 22 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:30.503977 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100f9d30-237b-4c2f-8e66-39678ae595cd None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:30.504501 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100f9d30-237b-4c2f-8e66-39678ae595cd None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:30.504994 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100f9d30-237b-4c2f-8e66-39678ae595cd None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:30.506604 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 5/10] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:32:30 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:32:30.514691 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3ef159-eacb-405f-b52e-23d4280045c7 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:30.515186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3ef159-eacb-405f-b52e-23d4280045c7 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:30.515346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3ef159-eacb-405f-b52e-23d4280045c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:30.534434 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-5c3ef159-eacb-405f-b52e-23d4280045c7 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:32:30.535677 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-5c3ef159-eacb-405f-b52e-23d4280045c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:32:30.538697 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a6554e-c6a1-4d4e-b8ce-5943e99e6f4e None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:30.538980 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a6554e-c6a1-4d4e-b8ce-5943e99e6f4e None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:30.539259 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a6554e-c6a1-4d4e-b8ce-5943e99e6f4e None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:30.540613 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 6/11] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:32:30 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:32:30.546136 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44ed1a3-60de-456b-a405-ebd71a4b011f None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:30.546423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44ed1a3-60de-456b-a405-ebd71a4b011f None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:30.546695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44ed1a3-60de-456b-a405-ebd71a4b011f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:30.589619 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG passlib.handlers.bcrypt [None req-f44ed1a3-60de-456b-a405-ebd71a4b011f None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=88218) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Aug 24 15:32:30.590044 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG passlib.handlers.bcrypt [None req-f44ed1a3-60de-456b-a405-ebd71a4b011f None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=88218) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Aug 24 15:32:30.620220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 6/12] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:32:30 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:32:30.636028 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-f44ed1a3-60de-456b-a405-ebd71a4b011f None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:32:30.637641 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-f44ed1a3-60de-456b-a405-ebd71a4b011f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:32:30.767008 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 7/13] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:32:30 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 15:32:30.900672 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3ef159-eacb-405f-b52e-23d4280045c7 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:30.927152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:31.013000 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3ef159-eacb-405f-b52e-23d4280045c7 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:31.043667 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44ed1a3-60de-456b-a405-ebd71a4b011f None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:31.048232 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c50e83-6744-4a22-a215-bbc0be069ba8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:31.048878 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c50e83-6744-4a22-a215-bbc0be069ba8 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:31.049163 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c50e83-6744-4a22-a215-bbc0be069ba8 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:31.049442 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c50e83-6744-4a22-a215-bbc0be069ba8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:31.061642 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c50e83-6744-4a22-a215-bbc0be069ba8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:32:31.071409 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-54c50e83-6744-4a22-a215-bbc0be069ba8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:32:31.071409 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:32:31.071409 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:32:31.071409 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:32:31.071409 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:32:31.071409 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:32:31.071409 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:32:31.071409 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:32:31.071409 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:32:31.071409 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:32:31.071409 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:32:31.071409 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:32:31.071409 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:32:31.071409 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:32:31.071409 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:32:31.071409 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:32:31.072445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:32:31.072445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:32:31.072445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:32:31.072445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:32:31.072445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:32:31.072445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:32:31.072445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:32:31.072445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:32:31.072445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:32:31.072445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:32:31.072445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:32:31.072445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:32:31.072445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:32:31.072445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:32:31.072445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:32:31.072445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:32:31.073238 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:32:31.073238 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:32:31.073238 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:32:31.073238 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:32:31.073238 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:32:31.073238 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:32:31.073238 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:32:31.073238 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:32:31.073238 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:32:31.073238 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:32:31.073238 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:32:31.073238 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:32:31.073238 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 7/14] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:32:30 2022] GET /identity/v3/roles/admin => generated 83 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:32:31.078570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c50e83-6744-4a22-a215-bbc0be069ba8 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:31.153805 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44ed1a3-60de-456b-a405-ebd71a4b011f None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:31.157200 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:31.168778 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:31.188090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c50e83-6744-4a22-a215-bbc0be069ba8 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:31.194770 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe79004-ff92-49d1-b23e-be2e2c82cbf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:31.195656 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe79004-ff92-49d1-b23e-be2e2c82cbf6 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:31.195977 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe79004-ff92-49d1-b23e-be2e2c82cbf6 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:31.196257 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe79004-ff92-49d1-b23e-be2e2c82cbf6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:31.231879 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ef8644-965d-4d03-a3c1-f73b55a0d14b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:31.232491 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ef8644-965d-4d03-a3c1-f73b55a0d14b None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:31.232768 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ef8644-965d-4d03-a3c1-f73b55a0d14b None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:31.233038 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ef8644-965d-4d03-a3c1-f73b55a0d14b None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:31.233802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ef8644-965d-4d03-a3c1-f73b55a0d14b None admin] RBAC: Adding query filter params (name=admin) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:32:31.234011 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ef8644-965d-4d03-a3c1-f73b55a0d14b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:31.235409 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ef8644-965d-4d03-a3c1-f73b55a0d14b None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:31.248222 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 8/15] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:32:31 2022] GET /identity/v3/roles?name=admin => generated 341 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:32:31.253743 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ef8644-965d-4d03-a3c1-f73b55a0d14b None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:31.269725 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe79004-ff92-49d1-b23e-be2e2c82cbf6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:31.271368 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe79004-ff92-49d1-b23e-be2e2c82cbf6 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:31.364961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ef8644-965d-4d03-a3c1-f73b55a0d14b None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:31.379714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 8/16] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:32:31 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 24 15:32:31.386241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe79004-ff92-49d1-b23e-be2e2c82cbf6 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:31.389391 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:31.402214 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a5d78b-0f63-4453-9001-0a16950a98b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:31.402949 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a5d78b-0f63-4453-9001-0a16950a98b3 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:31.403239 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a5d78b-0f63-4453-9001-0a16950a98b3 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:31.403513 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a5d78b-0f63-4453-9001-0a16950a98b3 None admin] PATH_INFO: `/v3/users/3eb7904f17244b85baf19c438c53917f` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:31.404716 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a5d78b-0f63-4453-9001-0a16950a98b3 None admin] RBAC: Authorizing `identity:get_user(user_id=3eb7904f17244b85baf19c438c53917f)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:31.406196 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a5d78b-0f63-4453-9001-0a16950a98b3 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:31.407820 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 9/17] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:32:31 2022] GET /identity/v3/users/3eb7904f17244b85baf19c438c53917f => generated 248 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:32:31.413645 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a5d78b-0f63-4453-9001-0a16950a98b3 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:31.430606 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:31.491962 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe79004-ff92-49d1-b23e-be2e2c82cbf6 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:31.518701 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a5d78b-0f63-4453-9001-0a16950a98b3 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:31.527378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af20fc72-d999-41e6-a49a-997e6e28b439 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:31.528262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af20fc72-d999-41e6-a49a-997e6e28b439 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:31.528548 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af20fc72-d999-41e6-a49a-997e6e28b439 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:31.528822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af20fc72-d999-41e6-a49a-997e6e28b439 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:31.548356 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af20fc72-d999-41e6-a49a-997e6e28b439 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:31.549896 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af20fc72-d999-41e6-a49a-997e6e28b439 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:31.554355 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3865113-555f-45cf-bea0-9cc0ff584897 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:31.554997 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3865113-555f-45cf-bea0-9cc0ff584897 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:31.555280 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3865113-555f-45cf-bea0-9cc0ff584897 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:31.555554 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3865113-555f-45cf-bea0-9cc0ff584897 None admin] PATH_INFO: `/v3/domains/default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:31.556714 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3865113-555f-45cf-bea0-9cc0ff584897 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:31.558174 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3865113-555f-45cf-bea0-9cc0ff584897 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:31.559778 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 10/18] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:32:31 2022] GET /identity/v3/domains/default => generated 200 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:32:31.566670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3865113-555f-45cf-bea0-9cc0ff584897 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:31.569945 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-af20fc72-d999-41e6-a49a-997e6e28b439 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:32:31.569945 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:32:31.569945 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:32:31.569945 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:32:31.569945 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:32:31.569945 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:32:31.569945 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:32:31.569945 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:32:31.569945 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:32:31.569945 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:32:31.569945 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:32:31.569945 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:32:31.569945 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:32:31.569945 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:32:31.569945 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:32:31.569945 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:32:31.570863 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:32:31.570863 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:32:31.570863 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:32:31.570863 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:32:31.570863 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:32:31.570863 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:32:31.570863 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:32:31.570863 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:32:31.570863 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:32:31.570863 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:32:31.570863 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:32:31.570863 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:32:31.570863 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:32:31.570863 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:32:31.570863 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:32:31.570863 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:32:31.570863 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:32:31.571687 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:32:31.571687 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:32:31.571687 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:32:31.571687 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 9/19] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:32:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:32:31.577357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af20fc72-d999-41e6-a49a-997e6e28b439 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:31.636428 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:31.651149 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:31.671885 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3865113-555f-45cf-bea0-9cc0ff584897 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:31.684568 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af20fc72-d999-41e6-a49a-997e6e28b439 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:31.707125 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1050a997-029d-4820-83ac-41f8ff1cec37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:31.707831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1050a997-029d-4820-83ac-41f8ff1cec37 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:31.708111 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1050a997-029d-4820-83ac-41f8ff1cec37 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:31.708384 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1050a997-029d-4820-83ac-41f8ff1cec37 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:31.709122 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1050a997-029d-4820-83ac-41f8ff1cec37 None admin] RBAC: Adding query filter params (role.id=4d8fbbaea81a4145b89af12350bb067c, scope.domain.id=default, user.id=3eb7904f17244b85baf19c438c53917f) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:32:31.709428 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1050a997-029d-4820-83ac-41f8ff1cec37 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:31.710900 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1050a997-029d-4820-83ac-41f8ff1cec37 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:31.718008 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 11/20] 10.222.0.102 () {56 vars in 1327 bytes} [Wed Aug 24 15:32:31 2022] GET /identity/v3/role_assignments?user.id=3eb7904f17244b85baf19c438c53917f&scope.domain.id=default&role.id=4d8fbbaea81a4145b89af12350bb067c => generated 234 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:32:31.719771 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab67ba8-2145-481a-871f-8f7377ab0420 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:31.720382 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab67ba8-2145-481a-871f-8f7377ab0420 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:31.720664 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab67ba8-2145-481a-871f-8f7377ab0420 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:31.720944 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab67ba8-2145-481a-871f-8f7377ab0420 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:31.721615 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab67ba8-2145-481a-871f-8f7377ab0420 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:32:31.721921 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab67ba8-2145-481a-871f-8f7377ab0420 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:31.724518 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab67ba8-2145-481a-871f-8f7377ab0420 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:31.745796 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 10/21] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:32:31 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:32:34.567675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c597417b-59ed-48e4-a335-f9abf29e5ebf None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:34.567675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c597417b-59ed-48e4-a335-f9abf29e5ebf None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:34.567675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c597417b-59ed-48e4-a335-f9abf29e5ebf None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:34.568950 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 12/22] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:32:34 2022] GET /identity => generated 271 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:32:34.586660 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b65200-eec2-4635-ac6f-af3167405c59 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:34.587244 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b65200-eec2-4635-ac6f-af3167405c59 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:34.587244 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b65200-eec2-4635-ac6f-af3167405c59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:34.624150 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-30b65200-eec2-4635-ac6f-af3167405c59 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:32:34.627657 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-30b65200-eec2-4635-ac6f-af3167405c59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:32:34.783669 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 11/23] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:32:34 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:32:34.820872 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9202e55-6a41-4065-9c38-ec577a40db78 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:34.821760 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9202e55-6a41-4065-9c38-ec577a40db78 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:34.822437 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9202e55-6a41-4065-9c38-ec577a40db78 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:34.824053 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 13/24] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:32:34 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:32:34.833164 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0634ecc-adc3-428d-ad76-89efc6214551 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:34.833449 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0634ecc-adc3-428d-ad76-89efc6214551 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:34.833732 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0634ecc-adc3-428d-ad76-89efc6214551 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:34.872991 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-b0634ecc-adc3-428d-ad76-89efc6214551 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:32:34.874309 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-b0634ecc-adc3-428d-ad76-89efc6214551 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:32:34.901696 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f752cc-3cc7-4270-9d20-94e08a566e1c None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:34.902382 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f752cc-3cc7-4270-9d20-94e08a566e1c None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:34.902857 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f752cc-3cc7-4270-9d20-94e08a566e1c None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:34.904573 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 14/25] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:32:34 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:32:34.915828 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c447942-4671-4ff4-be67-ea675c0983c6 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:34.916434 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c447942-4671-4ff4-be67-ea675c0983c6 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:34.916945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c447942-4671-4ff4-be67-ea675c0983c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:34.973875 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-7c447942-4671-4ff4-be67-ea675c0983c6 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:32:34.991675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-7c447942-4671-4ff4-be67-ea675c0983c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:32:35.017783 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:35.148713 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 12/26] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:32:34 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 317 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:32:35.282003 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 15/27] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:32:34 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 368 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 15:32:35.515762 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0634ecc-adc3-428d-ad76-89efc6214551 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:35.687612 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c447942-4671-4ff4-be67-ea675c0983c6 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:35.843695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0634ecc-adc3-428d-ad76-89efc6214551 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:35.947668 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:35.961487 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4696fda5-dd69-4d86-b189-5ec74b503b87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:35.962150 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4696fda5-dd69-4d86-b189-5ec74b503b87 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:35.962431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4696fda5-dd69-4d86-b189-5ec74b503b87 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:35.962703 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4696fda5-dd69-4d86-b189-5ec74b503b87 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:35.995259 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4696fda5-dd69-4d86-b189-5ec74b503b87 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:32:36.002428 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-4696fda5-dd69-4d86-b189-5ec74b503b87 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:32:36.002428 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:32:36.002428 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:32:36.002428 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:32:36.002428 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:32:36.002428 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:32:36.002428 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:32:36.002428 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:32:36.002428 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:32:36.002428 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:32:36.002428 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:32:36.002428 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:32:36.002428 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:32:36.002428 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:32:36.002428 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:32:36.002428 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:32:36.003365 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:32:36.003365 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:32:36.003365 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:32:36.003365 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:32:36.003365 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:32:36.003365 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:32:36.003365 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:32:36.003365 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:32:36.003365 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:32:36.003365 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:32:36.003365 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:32:36.003365 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:32:36.003365 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:32:36.003365 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:32:36.003365 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:32:36.003365 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:32:36.004683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:32:36.004683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:32:36.004683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:32:36.004683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:32:36.004683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:32:36.004683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:32:36.004683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:32:36.004683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:32:36.004683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:32:36.004683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:32:36.004683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:32:36.004683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:32:36.013225 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 13/28] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:32:35 2022] GET /identity/v3/roles/admin => generated 83 bytes in 499 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:32:36.014321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4696fda5-dd69-4d86-b189-5ec74b503b87 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:36.022367 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c447942-4671-4ff4-be67-ea675c0983c6 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:36.105407 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835c6e77-03f0-4e9c-98f7-593fceb942e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:36.106381 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835c6e77-03f0-4e9c-98f7-593fceb942e2 None admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:36.106824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835c6e77-03f0-4e9c-98f7-593fceb942e2 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:36.107238 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835c6e77-03f0-4e9c-98f7-593fceb942e2 None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:36.109985 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-835c6e77-03f0-4e9c-98f7-593fceb942e2 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:36.112003 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-835c6e77-03f0-4e9c-98f7-593fceb942e2 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:36.124906 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:36.252284 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4696fda5-dd69-4d86-b189-5ec74b503b87 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:36.302673 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a84762-923a-4de2-b0cc-9b8067312884 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:36.303544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a84762-923a-4de2-b0cc-9b8067312884 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:36.304068 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a84762-923a-4de2-b0cc-9b8067312884 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:36.304563 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a84762-923a-4de2-b0cc-9b8067312884 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:36.317126 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a84762-923a-4de2-b0cc-9b8067312884 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:36.318890 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a84762-923a-4de2-b0cc-9b8067312884 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:36.341308 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-35a84762-923a-4de2-b0cc-9b8067312884 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:32:36.341308 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:32:36.341308 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:32:36.341308 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:32:36.341308 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:32:36.341308 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:32:36.341308 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:32:36.341308 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:32:36.341308 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:32:36.341308 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:32:36.341308 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:32:36.341308 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:32:36.341308 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:32:36.341308 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:32:36.341308 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:32:36.341308 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:32:36.342246 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:32:36.342246 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:32:36.342246 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:32:36.342246 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:32:36.342246 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:32:36.342246 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:32:36.342246 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:32:36.342246 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:32:36.342246 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:32:36.342246 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:32:36.342246 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:32:36.342246 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:32:36.342246 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:32:36.342246 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:32:36.342246 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:32:36.342246 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:32:36.342246 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:32:36.343037 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:32:36.343037 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:32:36.343037 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:32:36.345959 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 14/29] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:32:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 332 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:32:36.346998 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a84762-923a-4de2-b0cc-9b8067312884 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:36.357930 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 16/30] 10.222.0.102 () {60 vars in 1136 bytes} [Wed Aug 24 15:32:35 2022] POST /identity/v3/roles => generated 220 bytes in 670 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 15:32:36.368415 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab22b1d-3b12-4f59-a0e7-4865d14a96f1 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:36.368712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab22b1d-3b12-4f59-a0e7-4865d14a96f1 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:36.369010 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab22b1d-3b12-4f59-a0e7-4865d14a96f1 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:36.375365 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 17/31] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:32:36 2022] GET /identity => generated 271 bytes in 13 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:32:36.380825 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39bab7e-af03-44fc-92ce-8c8a4d7e53f5 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:36.380825 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39bab7e-af03-44fc-92ce-8c8a4d7e53f5 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:36.381090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39bab7e-af03-44fc-92ce-8c8a4d7e53f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:36.448890 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-c39bab7e-af03-44fc-92ce-8c8a4d7e53f5 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:32:36.450507 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-c39bab7e-af03-44fc-92ce-8c8a4d7e53f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:32:36.478885 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:36.495894 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a84762-923a-4de2-b0cc-9b8067312884 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:36.555255 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8f2520-95b5-41be-94dd-b5ebf1571af8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:36.556134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8f2520-95b5-41be-94dd-b5ebf1571af8 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:36.556516 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8f2520-95b5-41be-94dd-b5ebf1571af8 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:36.556936 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8f2520-95b5-41be-94dd-b5ebf1571af8 None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:36.557729 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8f2520-95b5-41be-94dd-b5ebf1571af8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:32:36.558152 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8f2520-95b5-41be-94dd-b5ebf1571af8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:36.559695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8f2520-95b5-41be-94dd-b5ebf1571af8 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:36.574922 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 15/32] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:32:36 2022] GET /identity/v3/roles?name=admin => generated 341 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:36.575824 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8f2520-95b5-41be-94dd-b5ebf1571af8 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:36.679774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 18/33] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:32:36 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 294 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:32:36.679774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39bab7e-af03-44fc-92ce-8c8a4d7e53f5 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:36.718994 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8f2520-95b5-41be-94dd-b5ebf1571af8 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:36.762360 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:36.781306 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff34e6be-936b-407b-b73a-cd1b1e3c6181 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:36.782294 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff34e6be-936b-407b-b73a-cd1b1e3c6181 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:36.782583 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff34e6be-936b-407b-b73a-cd1b1e3c6181 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:36.782856 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff34e6be-936b-407b-b73a-cd1b1e3c6181 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:36.783535 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff34e6be-936b-407b-b73a-cd1b1e3c6181 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:32:36.790543 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff34e6be-936b-407b-b73a-cd1b1e3c6181 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:36.790543 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff34e6be-936b-407b-b73a-cd1b1e3c6181 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:36.803068 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 16/34] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:32:36 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:36.806103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653c4cc0-3e42-423f-bcae-baa6d0f200dc None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:36.806458 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653c4cc0-3e42-423f-bcae-baa6d0f200dc None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:36.806740 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653c4cc0-3e42-423f-bcae-baa6d0f200dc None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:36.808301 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 17/35] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:32:36 2022] GET /identity => generated 271 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:32:36.814980 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653c4cc0-3e42-423f-bcae-baa6d0f200dc None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:36.835312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:36.855369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39bab7e-af03-44fc-92ce-8c8a4d7e53f5 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:36.912466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2ef03f-24f2-443e-a91b-a5350cd75285 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:36.913464 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2ef03f-24f2-443e-a91b-a5350cd75285 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:36.913904 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2ef03f-24f2-443e-a91b-a5350cd75285 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:36.916560 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2ef03f-24f2-443e-a91b-a5350cd75285 None admin] PATH_INFO: `/v3/users/3eb7904f17244b85baf19c438c53917f` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:36.919164 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2ef03f-24f2-443e-a91b-a5350cd75285 None admin] RBAC: Authorizing `identity:get_user(user_id=3eb7904f17244b85baf19c438c53917f)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:36.922232 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2ef03f-24f2-443e-a91b-a5350cd75285 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:36.923324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 19/36] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:32:36 2022] GET /identity/v3/users/3eb7904f17244b85baf19c438c53917f => generated 248 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:36.924625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a629675c-0f87-4da4-b0e6-18f2d5c03c7e None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:36.924941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a629675c-0f87-4da4-b0e6-18f2d5c03c7e None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:36.925388 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a629675c-0f87-4da4-b0e6-18f2d5c03c7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:36.952946 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-a629675c-0f87-4da4-b0e6-18f2d5c03c7e None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:32:36.955059 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-a629675c-0f87-4da4-b0e6-18f2d5c03c7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:32:36.964154 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653c4cc0-3e42-423f-bcae-baa6d0f200dc None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:37.013031 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14321b39-2336-4b0c-85d6-2cabb791ded8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:37.013741 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14321b39-2336-4b0c-85d6-2cabb791ded8 None admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:37.014031 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14321b39-2336-4b0c-85d6-2cabb791ded8 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:37.014311 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14321b39-2336-4b0c-85d6-2cabb791ded8 None admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:37.015245 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14321b39-2336-4b0c-85d6-2cabb791ded8 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:37.016776 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14321b39-2336-4b0c-85d6-2cabb791ded8 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:37.066400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 20/37] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:32:36 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:32:37.068119 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ffddf9-95cb-4591-a6d0-b00b23264b4c None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:37.068588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ffddf9-95cb-4591-a6d0-b00b23264b4c None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:37.069057 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ffddf9-95cb-4591-a6d0-b00b23264b4c None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:37.070532 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 21/38] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:32:37 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:32:37.071392 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ffddf9-95cb-4591-a6d0-b00b23264b4c None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:37.170565 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:37.178693 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 18/39] 10.222.0.102 () {60 vars in 1146 bytes} [Wed Aug 24 15:32:36 2022] POST /identity/v3/projects => generated 302 bytes in 364 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:32:37.179884 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14321b39-2336-4b0c-85d6-2cabb791ded8 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:37.227713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ffddf9-95cb-4591-a6d0-b00b23264b4c None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:37.290154 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3a3735-50d6-49ca-a62d-bdc8bf0d1eeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:37.291182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3a3735-50d6-49ca-a62d-bdc8bf0d1eeb None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:37.291984 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3a3735-50d6-49ca-a62d-bdc8bf0d1eeb None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:37.292484 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3a3735-50d6-49ca-a62d-bdc8bf0d1eeb None admin] PATH_INFO: `/v3/domains/default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:37.294192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce3a3735-50d6-49ca-a62d-bdc8bf0d1eeb None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:37.296108 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce3a3735-50d6-49ca-a62d-bdc8bf0d1eeb None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:37.301471 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 22/40] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:32:37 2022] GET /identity/v3/domains/default => generated 200 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:37.303130 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4babd571-deec-4169-b6c5-f42b864b3558 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:37.303806 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4babd571-deec-4169-b6c5-f42b864b3558 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:37.304337 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4babd571-deec-4169-b6c5-f42b864b3558 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:37.329222 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:37.333678 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-4babd571-deec-4169-b6c5-f42b864b3558 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:32:37.335867 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-4babd571-deec-4169-b6c5-f42b864b3558 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:32:37.404387 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14321b39-2336-4b0c-85d6-2cabb791ded8 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:37.460341 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 23/41] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:32:37 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:32:37.461035 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4babd571-deec-4169-b6c5-f42b864b3558 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:37.470787 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273f1a68-75de-412e-ae70-d8c13179b419 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:37.471466 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273f1a68-75de-412e-ae70-d8c13179b419 None admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:37.471941 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273f1a68-75de-412e-ae70-d8c13179b419 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:37.472240 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273f1a68-75de-412e-ae70-d8c13179b419 None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:37.473585 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273f1a68-75de-412e-ae70-d8c13179b419 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:37.476770 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273f1a68-75de-412e-ae70-d8c13179b419 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:37.514115 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 19/42] 10.222.0.102 () {60 vars in 1136 bytes} [Wed Aug 24 15:32:37 2022] POST /identity/v3/roles => generated 226 bytes in 335 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:32:37.515118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273f1a68-75de-412e-ae70-d8c13179b419 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:37.603034 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:37.681293 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4babd571-deec-4169-b6c5-f42b864b3558 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:37.728035 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273f1a68-75de-412e-ae70-d8c13179b419 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:37.736304 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae12922-2bfb-48be-abc9-5dc4f8070563 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:37.737055 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae12922-2bfb-48be-abc9-5dc4f8070563 None admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:37.737335 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae12922-2bfb-48be-abc9-5dc4f8070563 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:37.737740 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae12922-2bfb-48be-abc9-5dc4f8070563 None admin] PATH_INFO: `/v3/domains/default/users/3eb7904f17244b85baf19c438c53917f/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:37.743115 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae12922-2bfb-48be-abc9-5dc4f8070563 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=3eb7904f17244b85baf19c438c53917f, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:37.743115 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae12922-2bfb-48be-abc9-5dc4f8070563 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:37.775361 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 24/43] 10.222.0.102 () {58 vars in 1366 bytes} [Wed Aug 24 15:32:37 2022] PUT /identity/v3/domains/default/users/3eb7904f17244b85baf19c438c53917f/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 315 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:32:37.809627 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae12922-2bfb-48be-abc9-5dc4f8070563 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:37.898560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:37.931662 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f4e40c-2a72-4d54-897f-dc78b361e585 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:37.931662 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f4e40c-2a72-4d54-897f-dc78b361e585 None admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:37.931662 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f4e40c-2a72-4d54-897f-dc78b361e585 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:37.931662 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f4e40c-2a72-4d54-897f-dc78b361e585 None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:37.932115 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f4e40c-2a72-4d54-897f-dc78b361e585 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:37.932115 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f4e40c-2a72-4d54-897f-dc78b361e585 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:37.955659 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 20/44] 10.222.0.102 () {60 vars in 1136 bytes} [Wed Aug 24 15:32:37 2022] POST /identity/v3/roles => generated 224 bytes in 439 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:32:38.033357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae12922-2bfb-48be-abc9-5dc4f8070563 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:38.072102 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:38.094638 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0808d92-7497-4726-938b-130e36a06fca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:38.095364 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0808d92-7497-4726-938b-130e36a06fca None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:38.095713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0808d92-7497-4726-938b-130e36a06fca None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:38.095993 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0808d92-7497-4726-938b-130e36a06fca None admin] PATH_INFO: `/v3/domains/default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:38.097939 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0808d92-7497-4726-938b-130e36a06fca None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:38.099823 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0808d92-7497-4726-938b-130e36a06fca None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:38.101584 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 25/45] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:32:37 2022] GET /identity/v3/domains/default => generated 200 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:32:38.107863 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f4e40c-2a72-4d54-897f-dc78b361e585 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:38.253890 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f4e40c-2a72-4d54-897f-dc78b361e585 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:38.306565 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ade7eb1-5ac6-47dc-bc8d-035240c4bc37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:38.307485 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ade7eb1-5ac6-47dc-bc8d-035240c4bc37 None admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:38.307993 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ade7eb1-5ac6-47dc-bc8d-035240c4bc37 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:38.308401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ade7eb1-5ac6-47dc-bc8d-035240c4bc37 None admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:38.309226 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ade7eb1-5ac6-47dc-bc8d-035240c4bc37 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:38.310893 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ade7eb1-5ac6-47dc-bc8d-035240c4bc37 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:38.350579 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 21/46] 10.222.0.102 () {60 vars in 1146 bytes} [Wed Aug 24 15:32:38 2022] POST /identity/v3/projects => generated 313 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:32:39.877747 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1226ff-7a59-43a7-a462-2c08cbd80ed7 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:39.878535 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1226ff-7a59-43a7-a462-2c08cbd80ed7 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:39.878694 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1226ff-7a59-43a7-a462-2c08cbd80ed7 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:39.880549 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 26/47] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:32:39 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:32:39.888549 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b7d161-54c9-4474-905d-c78f03afe52e None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:39.889337 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b7d161-54c9-4474-905d-c78f03afe52e None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:39.889337 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b7d161-54c9-4474-905d-c78f03afe52e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:39.912596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-f8b7d161-54c9-4474-905d-c78f03afe52e None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:32:39.914883 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-f8b7d161-54c9-4474-905d-c78f03afe52e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:32:39.969632 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:40.033946 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 22/48] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:32:39 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:32:40.321300 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1226ff-7a59-43a7-a462-2c08cbd80ed7 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:40.459816 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1226ff-7a59-43a7-a462-2c08cbd80ed7 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:40.507319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f0fbf0-c2f2-44ec-8edb-2c40742e77b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:40.508530 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f0fbf0-c2f2-44ec-8edb-2c40742e77b0 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:40.508986 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f0fbf0-c2f2-44ec-8edb-2c40742e77b0 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:40.509466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f0fbf0-c2f2-44ec-8edb-2c40742e77b0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:40.531884 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f0fbf0-c2f2-44ec-8edb-2c40742e77b0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:32:40.542490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-99f0fbf0-c2f2-44ec-8edb-2c40742e77b0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:32:40.542490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:32:40.542490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:32:40.542490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:32:40.542490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:32:40.542490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:32:40.542490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:32:40.542490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:32:40.542490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:32:40.542490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:32:40.542490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:32:40.542490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:32:40.542490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:32:40.542490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:32:40.542490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:32:40.542490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:32:40.543402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:32:40.543402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:32:40.543402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:32:40.543402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:32:40.543402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:32:40.543402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:32:40.543402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:32:40.543402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:32:40.543402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:32:40.543402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:32:40.543402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:32:40.543402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:32:40.543402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:32:40.543402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:32:40.543402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:32:40.543402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:32:40.544240 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:32:40.544240 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:32:40.544240 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:32:40.544240 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:32:40.544240 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:32:40.544240 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:32:40.544240 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:32:40.544240 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:32:40.544240 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:32:40.544240 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:32:40.544240 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:32:40.544240 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:32:40.544798 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 27/49] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:32:40 2022] GET /identity/v3/roles/admin => generated 83 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:32:40.551051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b7d161-54c9-4474-905d-c78f03afe52e None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:40.647409 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219a4f6c-f67e-4a52-996a-e8de3a921f1e None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:40.647835 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219a4f6c-f67e-4a52-996a-e8de3a921f1e None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:40.649423 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219a4f6c-f67e-4a52-996a-e8de3a921f1e None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:40.649423 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 28/50] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:32:40 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:32:40.655448 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834dcd3b-a0ab-4e46-8c69-b36d74a073fe None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:40.655754 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834dcd3b-a0ab-4e46-8c69-b36d74a073fe None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:40.656033 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834dcd3b-a0ab-4e46-8c69-b36d74a073fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:40.702654 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b7d161-54c9-4474-905d-c78f03afe52e None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:40.759219 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31f670f-6c46-46f8-a0c7-c05d6723aa00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:40.760228 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31f670f-6c46-46f8-a0c7-c05d6723aa00 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:40.760739 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31f670f-6c46-46f8-a0c7-c05d6723aa00 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:40.761188 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31f670f-6c46-46f8-a0c7-c05d6723aa00 None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:40.762063 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31f670f-6c46-46f8-a0c7-c05d6723aa00 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:32:40.762384 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31f670f-6c46-46f8-a0c7-c05d6723aa00 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:40.764075 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31f670f-6c46-46f8-a0c7-c05d6723aa00 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:40.775662 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 23/51] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:32:40 2022] GET /identity/v3/roles?name=admin => generated 341 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:40.781565 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31f670f-6c46-46f8-a0c7-c05d6723aa00 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:40.789732 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-834dcd3b-a0ab-4e46-8c69-b36d74a073fe None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:32:40.790969 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-834dcd3b-a0ab-4e46-8c69-b36d74a073fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:32:40.820318 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:40.861898 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:40.895508 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 29/52] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:32:40 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 15:32:40.927793 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31f670f-6c46-46f8-a0c7-c05d6723aa00 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:40.968750 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c86e51-489c-4d59-abeb-983ef7fc6dca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:40.969644 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c86e51-489c-4d59-abeb-983ef7fc6dca None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:40.970041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c86e51-489c-4d59-abeb-983ef7fc6dca None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:40.970367 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c86e51-489c-4d59-abeb-983ef7fc6dca None admin] PATH_INFO: `/v3/users/3eb7904f17244b85baf19c438c53917f` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:40.972322 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c86e51-489c-4d59-abeb-983ef7fc6dca None admin] RBAC: Authorizing `identity:get_user(user_id=3eb7904f17244b85baf19c438c53917f)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:40.973940 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c86e51-489c-4d59-abeb-983ef7fc6dca None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:40.976292 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 24/53] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:32:40 2022] GET /identity/v3/users/3eb7904f17244b85baf19c438c53917f => generated 248 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:32:40.984081 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834dcd3b-a0ab-4e46-8c69-b36d74a073fe None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:41.131564 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834dcd3b-a0ab-4e46-8c69-b36d74a073fe None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:41.178609 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c86e51-489c-4d59-abeb-983ef7fc6dca None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:41.189718 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669e65fb-a33a-45e4-8de4-01717e119393 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:41.190971 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669e65fb-a33a-45e4-8de4-01717e119393 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:41.191253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669e65fb-a33a-45e4-8de4-01717e119393 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:41.191530 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669e65fb-a33a-45e4-8de4-01717e119393 None admin] PATH_INFO: `/v3/domains/default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:41.193402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-669e65fb-a33a-45e4-8de4-01717e119393 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:41.195309 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-669e65fb-a33a-45e4-8de4-01717e119393 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:41.197360 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 30/54] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:32:40 2022] GET /identity/v3/domains/default => generated 200 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:41.204101 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669e65fb-a33a-45e4-8de4-01717e119393 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:41.258103 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:41.293815 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:41.304217 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c86e51-489c-4d59-abeb-983ef7fc6dca None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:41.332158 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669e65fb-a33a-45e4-8de4-01717e119393 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:41.355099 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1c2e9e-a5a7-4bfe-ada8-f4ba9648b905 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:41.355807 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1c2e9e-a5a7-4bfe-ada8-f4ba9648b905 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:41.356096 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1c2e9e-a5a7-4bfe-ada8-f4ba9648b905 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:41.356375 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1c2e9e-a5a7-4bfe-ada8-f4ba9648b905 None admin] PATH_INFO: `/v3/domains/default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:41.357572 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1c2e9e-a5a7-4bfe-ada8-f4ba9648b905 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:41.359132 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1c2e9e-a5a7-4bfe-ada8-f4ba9648b905 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:41.360806 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 25/55] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:32:41 2022] GET /identity/v3/domains/default => generated 200 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:32:41.367546 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1c2e9e-a5a7-4bfe-ada8-f4ba9648b905 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:41.378770 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45c32af-f018-40da-8f4e-03f2e388cb0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:41.378986 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45c32af-f018-40da-8f4e-03f2e388cb0f None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:41.379408 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45c32af-f018-40da-8f4e-03f2e388cb0f None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:41.379497 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45c32af-f018-40da-8f4e-03f2e388cb0f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:41.380334 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45c32af-f018-40da-8f4e-03f2e388cb0f None admin] RBAC: Adding query filter params (role.id=4d8fbbaea81a4145b89af12350bb067c, scope.domain.id=default, user.id=3eb7904f17244b85baf19c438c53917f) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:32:41.380658 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45c32af-f018-40da-8f4e-03f2e388cb0f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:41.382354 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45c32af-f018-40da-8f4e-03f2e388cb0f None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:41.390636 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 31/56] 10.222.0.102 () {56 vars in 1327 bytes} [Wed Aug 24 15:32:41 2022] GET /identity/v3/role_assignments?user.id=3eb7904f17244b85baf19c438c53917f&scope.domain.id=default&role.id=4d8fbbaea81a4145b89af12350bb067c => generated 532 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:41.502372 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1c2e9e-a5a7-4bfe-ada8-f4ba9648b905 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:41.545650 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53749e4-5d0e-4e1a-91f9-dc71e90bc12e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:41.546283 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53749e4-5d0e-4e1a-91f9-dc71e90bc12e None admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:41.546571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53749e4-5d0e-4e1a-91f9-dc71e90bc12e None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:41.546852 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53749e4-5d0e-4e1a-91f9-dc71e90bc12e None admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:41.547530 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c53749e4-5d0e-4e1a-91f9-dc71e90bc12e None admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:41.549074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c53749e4-5d0e-4e1a-91f9-dc71e90bc12e None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:41.590253 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 26/57] 10.222.0.102 () {60 vars in 1145 bytes} [Wed Aug 24 15:32:41 2022] POST /identity/v3/projects => generated 299 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 15:32:43.623578 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3dbfcc-ec24-4aa1-a3ad-b2db7a6eea33 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:43.624116 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3dbfcc-ec24-4aa1-a3ad-b2db7a6eea33 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:43.624203 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3dbfcc-ec24-4aa1-a3ad-b2db7a6eea33 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:43.625528 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 32/58] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:32:43 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:32:43.631335 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9ef633-34ac-4da3-b748-1da9747e6114 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:43.631815 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9ef633-34ac-4da3-b748-1da9747e6114 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:43.631933 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9ef633-34ac-4da3-b748-1da9747e6114 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:43.650511 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-ba9ef633-34ac-4da3-b748-1da9747e6114 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:32:43.651780 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-ba9ef633-34ac-4da3-b748-1da9747e6114 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:32:43.662770 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:43.732997 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 27/59] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:32:43 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:32:44.014340 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3dbfcc-ec24-4aa1-a3ad-b2db7a6eea33 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:44.134447 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3dbfcc-ec24-4aa1-a3ad-b2db7a6eea33 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:44.150102 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:44.187598 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a5e863-cbfa-485e-bd80-0c8589fe24d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:44.188405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a5e863-cbfa-485e-bd80-0c8589fe24d0 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:44.189120 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a5e863-cbfa-485e-bd80-0c8589fe24d0 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:44.189653 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a5e863-cbfa-485e-bd80-0c8589fe24d0 None admin] PATH_INFO: `/v3/domains/default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:44.191090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a5e863-cbfa-485e-bd80-0c8589fe24d0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:44.192712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a5e863-cbfa-485e-bd80-0c8589fe24d0 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:44.194592 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 33/60] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:32:44 2022] GET /identity/v3/domains/default => generated 200 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:44.200273 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9ef633-34ac-4da3-b748-1da9747e6114 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:44.312230 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9ef633-34ac-4da3-b748-1da9747e6114 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:44.355692 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de10684d-171d-4d25-8856-29b3334ee1ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:44.356624 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de10684d-171d-4d25-8856-29b3334ee1ef None admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:44.357098 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de10684d-171d-4d25-8856-29b3334ee1ef None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:44.357557 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de10684d-171d-4d25-8856-29b3334ee1ef None admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:44.358720 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de10684d-171d-4d25-8856-29b3334ee1ef None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:44.361281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de10684d-171d-4d25-8856-29b3334ee1ef None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:44.396115 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:44.402244 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 28/61] 10.222.0.102 () {60 vars in 1137 bytes} [Wed Aug 24 15:32:44 2022] POST /identity/v3/users => generated 276 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:32:47.592534 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a6aebc-8b00-43cf-864a-d359cda3595c None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:47.593075 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a6aebc-8b00-43cf-864a-d359cda3595c None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:47.593075 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a6aebc-8b00-43cf-864a-d359cda3595c None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:47.594430 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 34/62] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:32:47 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:32:47.600720 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3f2250-074a-49c6-8c1f-884e998547e5 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:47.601177 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3f2250-074a-49c6-8c1f-884e998547e5 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:47.601367 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3f2250-074a-49c6-8c1f-884e998547e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:47.621592 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e733c49-53cf-418c-8f14-a6a2e4eae5f3 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:47.622208 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e733c49-53cf-418c-8f14-a6a2e4eae5f3 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:47.622686 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e733c49-53cf-418c-8f14-a6a2e4eae5f3 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:47.624267 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 35/63] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:32:47 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:32:47.625829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-bd3f2250-074a-49c6-8c1f-884e998547e5 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:32:47.627111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-bd3f2250-074a-49c6-8c1f-884e998547e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:32:47.630822 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f3202a-d6c7-405c-9248-c03283da31cd None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:47.631366 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f3202a-d6c7-405c-9248-c03283da31cd None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:47.631880 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f3202a-d6c7-405c-9248-c03283da31cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:47.663218 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-64f3202a-d6c7-405c-9248-c03283da31cd None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:32:47.664454 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-64f3202a-d6c7-405c-9248-c03283da31cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:32:47.762792 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 29/64] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:32:47 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:32:47.782127 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 36/65] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:32:47 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 15:32:47.806753 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aec875f-c5f3-45b2-8208-f5dfe6609792 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:47.807064 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aec875f-c5f3-45b2-8208-f5dfe6609792 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:47.807311 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aec875f-c5f3-45b2-8208-f5dfe6609792 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:47.808748 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 30/66] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:32:47 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:32:47.814941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a8d1f9-cec4-4c2a-8d47-344e3963d547 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:47.815231 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a8d1f9-cec4-4c2a-8d47-344e3963d547 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:47.815506 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a8d1f9-cec4-4c2a-8d47-344e3963d547 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:47.844419 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-b4a8d1f9-cec4-4c2a-8d47-344e3963d547 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:32:47.846358 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-b4a8d1f9-cec4-4c2a-8d47-344e3963d547 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:32:47.925371 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 37/67] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:32:47 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:32:48.138032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aec875f-c5f3-45b2-8208-f5dfe6609792 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:48.214735 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a8d1f9-cec4-4c2a-8d47-344e3963d547 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:48.259657 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aec875f-c5f3-45b2-8208-f5dfe6609792 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:48.265717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:48.294154 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:48.302197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80a6b6a-bf51-4a1a-8943-0e354a88d19f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:48.302845 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80a6b6a-bf51-4a1a-8943-0e354a88d19f None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:48.303129 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80a6b6a-bf51-4a1a-8943-0e354a88d19f None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:48.303677 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80a6b6a-bf51-4a1a-8943-0e354a88d19f None admin] PATH_INFO: `/v3/roles/member` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:48.318952 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80a6b6a-bf51-4a1a-8943-0e354a88d19f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:32:48.323960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-d80a6b6a-bf51-4a1a-8943-0e354a88d19f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:32:48.323960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:32:48.323960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:32:48.323960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:32:48.323960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:32:48.323960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:32:48.323960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:32:48.323960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:32:48.323960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:32:48.323960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:32:48.323960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:32:48.323960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:32:48.323960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:32:48.323960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:32:48.323960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:32:48.323960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:32:48.324824 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:32:48.324824 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:32:48.324824 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:32:48.324824 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:32:48.324824 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:32:48.324824 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:32:48.324824 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:32:48.324824 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:32:48.324824 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:32:48.324824 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:32:48.324824 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:32:48.324824 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:32:48.324824 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:32:48.324824 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:32:48.324824 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:32:48.324824 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:32:48.325596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:32:48.325596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:32:48.325596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:32:48.325596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:32:48.325596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:32:48.325596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:32:48.325596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:32:48.325596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:32:48.325596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:32:48.325596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:32:48.325596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:32:48.325596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:32:48.325596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 31/68] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:32:48 2022] GET /identity/v3/roles/member => generated 84 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:32:48.326227 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80a6b6a-bf51-4a1a-8943-0e354a88d19f None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:48.363168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a8d1f9-cec4-4c2a-8d47-344e3963d547 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:48.411082 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38dbad61-5d7b-4974-bc0e-22787af2ec9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:48.411655 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dbad61-5d7b-4974-bc0e-22787af2ec9e None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:48.411958 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dbad61-5d7b-4974-bc0e-22787af2ec9e None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:48.412231 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dbad61-5d7b-4974-bc0e-22787af2ec9e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:48.423077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38dbad61-5d7b-4974-bc0e-22787af2ec9e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:32:48.428730 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-38dbad61-5d7b-4974-bc0e-22787af2ec9e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:32:48.428730 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:32:48.428730 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:32:48.428730 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:32:48.428730 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:32:48.428730 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:32:48.428730 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:32:48.428730 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:32:48.428730 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:32:48.428730 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:32:48.428730 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:32:48.428730 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:32:48.428730 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:32:48.428730 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:32:48.428730 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:32:48.428730 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:32:48.429577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:32:48.429577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:32:48.429577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:32:48.429577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:32:48.429577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:32:48.429577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:32:48.429577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:32:48.429577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:32:48.429577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:32:48.429577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:32:48.429577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:32:48.429577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:32:48.429577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:32:48.429577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:32:48.429577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:32:48.429577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:32:48.430331 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:32:48.430331 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:32:48.430331 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:32:48.430331 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:32:48.430331 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:32:48.430331 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:32:48.430331 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:32:48.430331 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:32:48.430331 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:32:48.430331 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:32:48.430331 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:32:48.430331 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:32:48.430331 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 38/69] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:32:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:32:48.436408 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da91112b-0c3f-498b-8c59-77cadc31a370 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:48.436408 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da91112b-0c3f-498b-8c59-77cadc31a370 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:48.436408 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da91112b-0c3f-498b-8c59-77cadc31a370 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:48.436408 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 39/70] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:32:48 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:32:48.436408 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da91112b-0c3f-498b-8c59-77cadc31a370 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:48.456590 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80a6b6a-bf51-4a1a-8943-0e354a88d19f None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:48.492830 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36480f7-4ab6-46dd-bc9d-e09f5321ab3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:48.493467 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36480f7-4ab6-46dd-bc9d-e09f5321ab3d None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:48.493750 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36480f7-4ab6-46dd-bc9d-e09f5321ab3d None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:48.494024 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36480f7-4ab6-46dd-bc9d-e09f5321ab3d None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:48.504767 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a36480f7-4ab6-46dd-bc9d-e09f5321ab3d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:32:48.509491 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-a36480f7-4ab6-46dd-bc9d-e09f5321ab3d None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 15:32:48.509491 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:32:48.509491 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:32:48.509491 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:32:48.509491 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:32:48.509491 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:32:48.509491 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:32:48.509491 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:32:48.509491 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:32:48.509491 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:32:48.509491 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:32:48.509491 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:32:48.509491 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:32:48.509491 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:32:48.509491 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:32:48.509491 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:32:48.510320 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:32:48.510320 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:32:48.510320 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:32:48.510320 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:32:48.510320 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:32:48.510320 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:32:48.510320 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:32:48.510320 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:32:48.510320 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:32:48.510320 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:32:48.510320 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:32:48.510320 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:32:48.510320 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:32:48.510320 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:32:48.510320 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:32:48.510320 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:32:48.511174 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:32:48.511174 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:32:48.511174 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:32:48.511174 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:32:48.511174 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:32:48.511174 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:32:48.511174 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:32:48.511174 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:32:48.511174 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:32:48.511174 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:32:48.511174 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 15:32:48.511174 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:32:48.511174 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 32/71] 10.222.0.102 () {56 vars in 1119 bytes} [Wed Aug 24 15:32:48 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:32:48.513000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36480f7-4ab6-46dd-bc9d-e09f5321ab3d None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:48.546160 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:48.551683 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da91112b-0c3f-498b-8c59-77cadc31a370 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:48.571333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:48.586876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b877a08e-bea7-4e5a-a9d7-efed4f41e474 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:48.587492 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b877a08e-bea7-4e5a-a9d7-efed4f41e474 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:48.587810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b877a08e-bea7-4e5a-a9d7-efed4f41e474 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:48.588085 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b877a08e-bea7-4e5a-a9d7-efed4f41e474 None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:48.588759 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b877a08e-bea7-4e5a-a9d7-efed4f41e474 None admin] RBAC: Adding query filter params (name=member) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:32:48.589065 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b877a08e-bea7-4e5a-a9d7-efed4f41e474 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:48.590445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b877a08e-bea7-4e5a-a9d7-efed4f41e474 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:48.599570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 40/72] 10.222.0.102 () {56 vars in 1106 bytes} [Wed Aug 24 15:32:48 2022] GET /identity/v3/roles?name=member => generated 343 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:48.600808 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9be26a8-cbb7-48d7-850f-bf2b8465bc63 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:48.601098 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9be26a8-cbb7-48d7-850f-bf2b8465bc63 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:48.601469 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9be26a8-cbb7-48d7-850f-bf2b8465bc63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:48.620106 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-a9be26a8-cbb7-48d7-850f-bf2b8465bc63 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:32:48.621325 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-a9be26a8-cbb7-48d7-850f-bf2b8465bc63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:32:48.630029 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36480f7-4ab6-46dd-bc9d-e09f5321ab3d None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:48.669295 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b95f8b-aed2-4c4c-af22-f7079390f85d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:48.669826 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b95f8b-aed2-4c4c-af22-f7079390f85d None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:48.670114 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b95f8b-aed2-4c4c-af22-f7079390f85d None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:48.670396 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b95f8b-aed2-4c4c-af22-f7079390f85d None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:48.671096 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b95f8b-aed2-4c4c-af22-f7079390f85d None admin] RBAC: Adding query filter params (name=admin) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:32:48.671411 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b95f8b-aed2-4c4c-af22-f7079390f85d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:48.672854 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b95f8b-aed2-4c4c-af22-f7079390f85d None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:48.682422 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 33/73] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:32:48 2022] GET /identity/v3/roles?name=admin => generated 341 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:48.683748 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c3d1d0-09c0-416a-bec5-81951de124b0 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:48.684036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c3d1d0-09c0-416a-bec5-81951de124b0 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:48.684321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c3d1d0-09c0-416a-bec5-81951de124b0 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:48.687801 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 34/74] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:32:48 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:32:48.687801 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c3d1d0-09c0-416a-bec5-81951de124b0 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:48.695486 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 41/75] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:32:48 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:32:48.705157 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9be26a8-cbb7-48d7-850f-bf2b8465bc63 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:48.828042 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:48.842138 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c3d1d0-09c0-416a-bec5-81951de124b0 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:48.851605 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9be26a8-cbb7-48d7-850f-bf2b8465bc63 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:48.867240 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:48.896159 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc039f0e-7ac4-4a38-ad59-e67edd25716f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:48.896659 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1828dd-3a3f-485d-aa80-22eca308f293 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:48.896802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc039f0e-7ac4-4a38-ad59-e67edd25716f None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:48.897087 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc039f0e-7ac4-4a38-ad59-e67edd25716f None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:48.897326 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1828dd-3a3f-485d-aa80-22eca308f293 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:48.897485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc039f0e-7ac4-4a38-ad59-e67edd25716f None admin] PATH_INFO: `/v3/users/cb1c2ffc97364fa48e0d109484b7871c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:48.897608 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1828dd-3a3f-485d-aa80-22eca308f293 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:48.897842 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1828dd-3a3f-485d-aa80-22eca308f293 None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:48.898520 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1828dd-3a3f-485d-aa80-22eca308f293 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:32:48.898834 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1828dd-3a3f-485d-aa80-22eca308f293 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:48.900276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1828dd-3a3f-485d-aa80-22eca308f293 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:48.908855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 35/76] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:32:48 2022] GET /identity/v3/roles?name=anotherrole => generated 336 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:48.909687 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1828dd-3a3f-485d-aa80-22eca308f293 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:48.922954 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc039f0e-7ac4-4a38-ad59-e67edd25716f None admin] RBAC: Authorizing `identity:get_user(user_id=cb1c2ffc97364fa48e0d109484b7871c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:48.924682 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc039f0e-7ac4-4a38-ad59-e67edd25716f None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:48.926585 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 42/77] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:32:48 2022] GET /identity/v3/users/cb1c2ffc97364fa48e0d109484b7871c => generated 276 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:48.927605 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e382263-9ccb-44f0-904e-1b7c3ede4d65 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:48.927921 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e382263-9ccb-44f0-904e-1b7c3ede4d65 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:48.928203 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e382263-9ccb-44f0-904e-1b7c3ede4d65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:48.952963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-6e382263-9ccb-44f0-904e-1b7c3ede4d65 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:32:48.954572 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-6e382263-9ccb-44f0-904e-1b7c3ede4d65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:32:49.029127 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 43/78] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:32:48 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:32:49.030240 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1828dd-3a3f-485d-aa80-22eca308f293 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:49.030511 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e382263-9ccb-44f0-904e-1b7c3ede4d65 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:49.068754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6880585-3084-4af7-b24c-e968c66379e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:49.069393 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6880585-3084-4af7-b24c-e968c66379e7 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:49.069679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6880585-3084-4af7-b24c-e968c66379e7 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:49.070136 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6880585-3084-4af7-b24c-e968c66379e7 None admin] PATH_INFO: `/v3/users/3eb7904f17244b85baf19c438c53917f` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:49.071410 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6880585-3084-4af7-b24c-e968c66379e7 None admin] RBAC: Authorizing `identity:get_user(user_id=3eb7904f17244b85baf19c438c53917f)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:49.073253 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6880585-3084-4af7-b24c-e968c66379e7 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:49.082616 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 36/79] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:32:48 2022] GET /identity/v3/users/3eb7904f17244b85baf19c438c53917f => generated 248 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:49.082616 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6880585-3084-4af7-b24c-e968c66379e7 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:49.125561 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:49.134187 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:49.151479 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e382263-9ccb-44f0-904e-1b7c3ede4d65 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:49.195377 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589b7573-292f-46d2-8676-66182c6d1d27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:49.195975 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589b7573-292f-46d2-8676-66182c6d1d27 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:49.196257 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589b7573-292f-46d2-8676-66182c6d1d27 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:49.196537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589b7573-292f-46d2-8676-66182c6d1d27 None admin] PATH_INFO: `/v3/users/cb1c2ffc97364fa48e0d109484b7871c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:49.197694 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-589b7573-292f-46d2-8676-66182c6d1d27 None admin] RBAC: Authorizing `identity:get_user(user_id=cb1c2ffc97364fa48e0d109484b7871c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:49.199178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-589b7573-292f-46d2-8676-66182c6d1d27 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:49.200962 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 44/80] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:32:49 2022] GET /identity/v3/users/cb1c2ffc97364fa48e0d109484b7871c => generated 276 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:49.201557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589b7573-292f-46d2-8676-66182c6d1d27 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:49.208960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6880585-3084-4af7-b24c-e968c66379e7 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:49.248762 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52d51ef-3c1a-438f-8e16-b8f23323f962 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:49.249477 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52d51ef-3c1a-438f-8e16-b8f23323f962 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:49.249761 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52d51ef-3c1a-438f-8e16-b8f23323f962 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:49.250033 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52d51ef-3c1a-438f-8e16-b8f23323f962 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:49.321212 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589b7573-292f-46d2-8676-66182c6d1d27 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:49.329947 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52d51ef-3c1a-438f-8e16-b8f23323f962 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:49.331403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52d51ef-3c1a-438f-8e16-b8f23323f962 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:49.357487 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4e32c1-8288-44b2-ad0f-3b88eb0baf60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:49.358131 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4e32c1-8288-44b2-ad0f-3b88eb0baf60 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:49.358416 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4e32c1-8288-44b2-ad0f-3b88eb0baf60 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:49.358695 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4e32c1-8288-44b2-ad0f-3b88eb0baf60 None admin] PATH_INFO: `/v3/roles/member` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:49.370702 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad4e32c1-8288-44b2-ad0f-3b88eb0baf60 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:32:49.375742 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-ad4e32c1-8288-44b2-ad0f-3b88eb0baf60 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:32:49.375742 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:32:49.375742 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:32:49.375742 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:32:49.375742 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:32:49.375742 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:32:49.375742 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:32:49.375742 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:32:49.375742 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:32:49.375742 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:32:49.375742 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:32:49.375742 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:32:49.375742 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:32:49.375742 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:32:49.375742 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:32:49.375742 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:32:49.377369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:32:49.377369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:32:49.377369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:32:49.377369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:32:49.377369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:32:49.377369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:32:49.377369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:32:49.377369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:32:49.377369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:32:49.377369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:32:49.377369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:32:49.377369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:32:49.377369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:32:49.377369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:32:49.377369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:32:49.377369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:32:49.378124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:32:49.378124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:32:49.378124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:32:49.378124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:32:49.378124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:32:49.378124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:32:49.378124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:32:49.378124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:32:49.378124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:32:49.378124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:32:49.378124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:32:49.378124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:32:49.378864 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 45/81] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:32:49 2022] GET /identity/v3/roles/member => generated 84 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:32:49.379091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4e32c1-8288-44b2-ad0f-3b88eb0baf60 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:49.379398 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:49.390691 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:49.440131 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 37/82] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:32:49 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:32:49.441194 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52d51ef-3c1a-438f-8e16-b8f23323f962 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:49.489090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4e32c1-8288-44b2-ad0f-3b88eb0baf60 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:49.526326 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ac36ec-17fc-4dd3-9819-4399b8aaa6ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:49.526941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ac36ec-17fc-4dd3-9819-4399b8aaa6ea None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:49.527224 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ac36ec-17fc-4dd3-9819-4399b8aaa6ea None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:49.527501 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ac36ec-17fc-4dd3-9819-4399b8aaa6ea None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:49.550421 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52d51ef-3c1a-438f-8e16-b8f23323f962 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:49.585554 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca78e72-4c05-49c0-9199-c80ad3d4c998 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:49.586159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca78e72-4c05-49c0-9199-c80ad3d4c998 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:49.586446 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca78e72-4c05-49c0-9199-c80ad3d4c998 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:49.586728 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca78e72-4c05-49c0-9199-c80ad3d4c998 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:49.598658 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ac36ec-17fc-4dd3-9819-4399b8aaa6ea None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:49.599593 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:49.600196 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ac36ec-17fc-4dd3-9819-4399b8aaa6ea None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:49.608201 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:49.657506 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca78e72-4c05-49c0-9199-c80ad3d4c998 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:49.659497 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca78e72-4c05-49c0-9199-c80ad3d4c998 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:49.707019 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 46/83] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:32:49 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:32:49.707924 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ac36ec-17fc-4dd3-9819-4399b8aaa6ea None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:49.768981 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 38/84] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:32:49 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:32:49.769745 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca78e72-4c05-49c0-9199-c80ad3d4c998 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:49.821522 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ac36ec-17fc-4dd3-9819-4399b8aaa6ea None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:49.821971 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:49.832947 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:49.867830 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5f78de-3100-425e-b763-3bbd5d971fa1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:49.868474 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5f78de-3100-425e-b763-3bbd5d971fa1 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:49.868890 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5f78de-3100-425e-b763-3bbd5d971fa1 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:49.869154 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5f78de-3100-425e-b763-3bbd5d971fa1 None admin] PATH_INFO: `/v3/domains/default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:49.870510 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b5f78de-3100-425e-b763-3bbd5d971fa1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:49.872032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b5f78de-3100-425e-b763-3bbd5d971fa1 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:49.873636 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 47/85] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:32:49 2022] GET /identity/v3/domains/default => generated 200 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:49.875770 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5f78de-3100-425e-b763-3bbd5d971fa1 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:49.901437 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca78e72-4c05-49c0-9199-c80ad3d4c998 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:49.936828 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8ff483-f293-45f9-af10-3cc9ee10048e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:49.937433 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8ff483-f293-45f9-af10-3cc9ee10048e None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:49.937717 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8ff483-f293-45f9-af10-3cc9ee10048e None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:49.937999 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8ff483-f293-45f9-af10-3cc9ee10048e None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:49.939002 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8ff483-f293-45f9-af10-3cc9ee10048e None admin] RBAC: Adding query filter params (name=member) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:32:49.939315 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8ff483-f293-45f9-af10-3cc9ee10048e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:49.940847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8ff483-f293-45f9-af10-3cc9ee10048e None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:49.949070 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 39/86] 10.222.0.102 () {56 vars in 1106 bytes} [Wed Aug 24 15:32:49 2022] GET /identity/v3/roles?name=member => generated 343 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:49.949884 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8ff483-f293-45f9-af10-3cc9ee10048e None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:49.977712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5f78de-3100-425e-b763-3bbd5d971fa1 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:50.017687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f079e3-754a-437c-a924-5cf5ae8bf737 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:50.018322 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f079e3-754a-437c-a924-5cf5ae8bf737 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:50.018607 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f079e3-754a-437c-a924-5cf5ae8bf737 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:50.018885 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f079e3-754a-437c-a924-5cf5ae8bf737 None admin] PATH_INFO: `/v3/projects/749c6d651fc445cdb75db25a73873386` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:50.020136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f079e3-754a-437c-a924-5cf5ae8bf737 None admin] RBAC: Authorizing `identity:get_project(project_id=749c6d651fc445cdb75db25a73873386)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:50.021542 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f079e3-754a-437c-a924-5cf5ae8bf737 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:50.023194 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 48/87] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:32:49 2022] GET /identity/v3/projects/749c6d651fc445cdb75db25a73873386 => generated 299 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:50.024034 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f079e3-754a-437c-a924-5cf5ae8bf737 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:50.071113 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8ff483-f293-45f9-af10-3cc9ee10048e None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:50.071939 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:50.073856 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:50.112062 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8052064-3700-427d-b814-8185fcf21368 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:50.112696 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8052064-3700-427d-b814-8185fcf21368 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:50.112990 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8052064-3700-427d-b814-8185fcf21368 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:50.113267 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8052064-3700-427d-b814-8185fcf21368 None admin] PATH_INFO: `/v3/projects/749c6d651fc445cdb75db25a73873386` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:50.114421 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8052064-3700-427d-b814-8185fcf21368 None admin] RBAC: Authorizing `identity:get_project(project_id=749c6d651fc445cdb75db25a73873386)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:50.115902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8052064-3700-427d-b814-8185fcf21368 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:50.117553 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 40/88] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:32:49 2022] GET /identity/v3/projects/749c6d651fc445cdb75db25a73873386 => generated 299 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:50.118316 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8052064-3700-427d-b814-8185fcf21368 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:50.135457 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f079e3-754a-437c-a924-5cf5ae8bf737 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:50.172317 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926ad426-300b-48e4-b8d9-d9c4c0d42353 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:50.172960 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926ad426-300b-48e4-b8d9-d9c4c0d42353 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:50.173276 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926ad426-300b-48e4-b8d9-d9c4c0d42353 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:50.173844 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926ad426-300b-48e4-b8d9-d9c4c0d42353 None admin] PATH_INFO: `/v3/projects/749c6d651fc445cdb75db25a73873386` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:50.174679 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926ad426-300b-48e4-b8d9-d9c4c0d42353 None admin] RBAC: Authorizing `identity:get_project(project_id=749c6d651fc445cdb75db25a73873386)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:50.176166 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926ad426-300b-48e4-b8d9-d9c4c0d42353 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:50.177822 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 49/89] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:32:50 2022] GET /identity/v3/projects/749c6d651fc445cdb75db25a73873386 => generated 299 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:50.183410 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926ad426-300b-48e4-b8d9-d9c4c0d42353 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:50.224390 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8052064-3700-427d-b814-8185fcf21368 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:50.259912 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7e6b4d-effe-42ac-ae01-4541726bed2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:50.260448 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7e6b4d-effe-42ac-ae01-4541726bed2b None admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:50.260732 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7e6b4d-effe-42ac-ae01-4541726bed2b None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:50.261007 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7e6b4d-effe-42ac-ae01-4541726bed2b None admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:50.261662 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f7e6b4d-effe-42ac-ae01-4541726bed2b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:50.263109 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f7e6b4d-effe-42ac-ae01-4541726bed2b None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:50.284574 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 41/90] 10.222.0.102 () {60 vars in 1137 bytes} [Wed Aug 24 15:32:50 2022] POST /identity/v3/users => generated 290 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:32:50.285325 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7e6b4d-effe-42ac-ae01-4541726bed2b None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:50.296604 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926ad426-300b-48e4-b8d9-d9c4c0d42353 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:50.313006 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:50.334907 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:50.345357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99371bfb-b666-4fc3-8a78-3399da9b982d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:50.346019 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99371bfb-b666-4fc3-8a78-3399da9b982d None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:50.346307 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99371bfb-b666-4fc3-8a78-3399da9b982d None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:50.346625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99371bfb-b666-4fc3-8a78-3399da9b982d None admin] PATH_INFO: `/v3/users/cb1c2ffc97364fa48e0d109484b7871c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:50.347831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99371bfb-b666-4fc3-8a78-3399da9b982d None admin] RBAC: Authorizing `identity:get_user(user_id=cb1c2ffc97364fa48e0d109484b7871c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:50.349319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99371bfb-b666-4fc3-8a78-3399da9b982d None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:50.351197 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 50/91] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:32:50 2022] GET /identity/v3/users/cb1c2ffc97364fa48e0d109484b7871c => generated 276 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:50.351872 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99371bfb-b666-4fc3-8a78-3399da9b982d None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:50.425996 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7e6b4d-effe-42ac-ae01-4541726bed2b None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:50.471457 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494da68f-8cae-452e-aad1-80587a73081a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:50.472089 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494da68f-8cae-452e-aad1-80587a73081a None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:50.472378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494da68f-8cae-452e-aad1-80587a73081a None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:50.472704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494da68f-8cae-452e-aad1-80587a73081a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:50.473457 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-494da68f-8cae-452e-aad1-80587a73081a None admin] RBAC: Adding query filter params (role.id=b5d2ed3e20ab47eeb741b4f43ab2eebb, scope.project.id=749c6d651fc445cdb75db25a73873386, user.id=cb1c2ffc97364fa48e0d109484b7871c) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:32:50.473677 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99371bfb-b666-4fc3-8a78-3399da9b982d None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:50.473781 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-494da68f-8cae-452e-aad1-80587a73081a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:50.475208 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-494da68f-8cae-452e-aad1-80587a73081a None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:50.489345 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 42/92] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:32:50 2022] GET /identity/v3/role_assignments?user.id=cb1c2ffc97364fa48e0d109484b7871c&scope.project.id=749c6d651fc445cdb75db25a73873386&role.id=b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 260 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:50.489345 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494da68f-8cae-452e-aad1-80587a73081a None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:50.522643 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1d24dd-9c27-45c8-9552-331e01ba6e53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:50.523315 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1d24dd-9c27-45c8-9552-331e01ba6e53 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:50.523602 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1d24dd-9c27-45c8-9552-331e01ba6e53 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:50.524012 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1d24dd-9c27-45c8-9552-331e01ba6e53 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:50.524784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1d24dd-9c27-45c8-9552-331e01ba6e53 None admin] RBAC: Adding query filter params (role.id=4d8fbbaea81a4145b89af12350bb067c, scope.project.id=749c6d651fc445cdb75db25a73873386, user.id=3eb7904f17244b85baf19c438c53917f) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:32:50.525102 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1d24dd-9c27-45c8-9552-331e01ba6e53 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:50.526633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1d24dd-9c27-45c8-9552-331e01ba6e53 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:50.536155 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 51/93] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:32:50 2022] GET /identity/v3/role_assignments?user.id=3eb7904f17244b85baf19c438c53917f&scope.project.id=749c6d651fc445cdb75db25a73873386&role.id=4d8fbbaea81a4145b89af12350bb067c => generated 260 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:50.536155 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1d24dd-9c27-45c8-9552-331e01ba6e53 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:50.597623 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:50.629845 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:50.647532 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494da68f-8cae-452e-aad1-80587a73081a None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:50.700159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f468247-09f1-4c23-b185-be6ab4c30e1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:50.701704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f468247-09f1-4c23-b185-be6ab4c30e1a None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:50.702195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f468247-09f1-4c23-b185-be6ab4c30e1a None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:50.704496 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f468247-09f1-4c23-b185-be6ab4c30e1a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:50.705575 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f468247-09f1-4c23-b185-be6ab4c30e1a None admin] RBAC: Adding query filter params (role.id=b8429ebd9c2a4cbbbbf30ad56ce23bfa, scope.project.id=749c6d651fc445cdb75db25a73873386, user.id=cb1c2ffc97364fa48e0d109484b7871c) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:32:50.705901 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f468247-09f1-4c23-b185-be6ab4c30e1a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:50.707429 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f468247-09f1-4c23-b185-be6ab4c30e1a None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:50.713932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 43/94] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:32:50 2022] GET /identity/v3/role_assignments?user.id=cb1c2ffc97364fa48e0d109484b7871c&scope.project.id=749c6d651fc445cdb75db25a73873386&role.id=b8429ebd9c2a4cbbbbf30ad56ce23bfa => generated 260 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:50.721022 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1d24dd-9c27-45c8-9552-331e01ba6e53 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:50.766441 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5891f7d9-104c-46b0-8c54-189fbd60bc08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:50.767507 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5891f7d9-104c-46b0-8c54-189fbd60bc08 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:50.768020 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5891f7d9-104c-46b0-8c54-189fbd60bc08 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:50.768495 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5891f7d9-104c-46b0-8c54-189fbd60bc08 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:50.855174 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5891f7d9-104c-46b0-8c54-189fbd60bc08 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:50.856732 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5891f7d9-104c-46b0-8c54-189fbd60bc08 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:50.907026 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:50.972674 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 52/95] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:32:50 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 438 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:32:50.980886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f468247-09f1-4c23-b185-be6ab4c30e1a None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:51.096044 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f468247-09f1-4c23-b185-be6ab4c30e1a None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:51.138123 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddf474e-854d-409e-b316-cf962f1d2cfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:51.138763 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddf474e-854d-409e-b316-cf962f1d2cfa None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:51.139049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddf474e-854d-409e-b316-cf962f1d2cfa None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:51.139326 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddf474e-854d-409e-b316-cf962f1d2cfa None admin] PATH_INFO: `/v3/projects/2dcca3c9f7d64026b4e21bfca99e2ab8` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:51.140622 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cddf474e-854d-409e-b316-cf962f1d2cfa None admin] RBAC: Authorizing `identity:get_project(project_id=2dcca3c9f7d64026b4e21bfca99e2ab8)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:51.142755 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cddf474e-854d-409e-b316-cf962f1d2cfa None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:51.145158 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 44/96] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:32:50 2022] GET /identity/v3/projects/2dcca3c9f7d64026b4e21bfca99e2ab8 => generated 313 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:51.151242 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5891f7d9-104c-46b0-8c54-189fbd60bc08 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:51.260517 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5891f7d9-104c-46b0-8c54-189fbd60bc08 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:51.298672 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a539b6d4-a646-4e97-b577-56ef4b2d1500 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:51.299324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a539b6d4-a646-4e97-b577-56ef4b2d1500 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:51.299734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a539b6d4-a646-4e97-b577-56ef4b2d1500 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:51.299984 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a539b6d4-a646-4e97-b577-56ef4b2d1500 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:51.300729 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a539b6d4-a646-4e97-b577-56ef4b2d1500 None admin] RBAC: Adding query filter params (role.id=b5d2ed3e20ab47eeb741b4f43ab2eebb, scope.project.id=2dcca3c9f7d64026b4e21bfca99e2ab8, user.id=cb1c2ffc97364fa48e0d109484b7871c) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:32:51.301045 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a539b6d4-a646-4e97-b577-56ef4b2d1500 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:51.302590 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a539b6d4-a646-4e97-b577-56ef4b2d1500 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:51.308683 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 53/97] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:32:51 2022] GET /identity/v3/role_assignments?user.id=cb1c2ffc97364fa48e0d109484b7871c&scope.project.id=2dcca3c9f7d64026b4e21bfca99e2ab8&role.id=b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 260 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:52.992007 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d55369-6bf6-470e-a752-680d6e7c4b37 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:52.992536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d55369-6bf6-470e-a752-680d6e7c4b37 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:52.992536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d55369-6bf6-470e-a752-680d6e7c4b37 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:52.993893 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 45/98] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:32:52 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:32:52.999269 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195fa7a7-aa5c-48c7-9130-a5e25ddf405b None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:53.000123 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195fa7a7-aa5c-48c7-9130-a5e25ddf405b None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:53.000591 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195fa7a7-aa5c-48c7-9130-a5e25ddf405b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:53.017719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:53.025181 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-195fa7a7-aa5c-48c7-9130-a5e25ddf405b None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:32:53.026402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-195fa7a7-aa5c-48c7-9130-a5e25ddf405b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:32:53.126587 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 54/99] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:32:52 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 15:32:53.196682 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27ee12a-9568-47e8-bc16-99775bac3991 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:53.197572 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27ee12a-9568-47e8-bc16-99775bac3991 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:53.198046 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27ee12a-9568-47e8-bc16-99775bac3991 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:53.199706 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 46/100] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:32:53 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:32:53.205999 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eaca14b-d3ab-49d0-a652-18688d1c2229 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:53.206284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eaca14b-d3ab-49d0-a652-18688d1c2229 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:53.206569 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eaca14b-d3ab-49d0-a652-18688d1c2229 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:53.233881 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-8eaca14b-d3ab-49d0-a652-18688d1c2229 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:32:53.235089 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-8eaca14b-d3ab-49d0-a652-18688d1c2229 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:32:53.332881 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 55/101] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:32:53 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:32:53.477049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fdb441-e80b-4e6b-bbd7-70b5b588a974 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:53.477679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fdb441-e80b-4e6b-bbd7-70b5b588a974 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:53.478193 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fdb441-e80b-4e6b-bbd7-70b5b588a974 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:53.479744 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 47/102] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:32:53 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:32:53.485148 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d9b482-b5f8-4d6c-af44-cd4bb3a8d89a None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:53.486223 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d9b482-b5f8-4d6c-af44-cd4bb3a8d89a None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:53.486589 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d9b482-b5f8-4d6c-af44-cd4bb3a8d89a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:53.514290 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-36d9b482-b5f8-4d6c-af44-cd4bb3a8d89a None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:32:53.515512 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-36d9b482-b5f8-4d6c-af44-cd4bb3a8d89a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:32:53.606388 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:53.615497 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 56/103] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:32:53 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:32:53.746438 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49fdb441-e80b-4e6b-bbd7-70b5b588a974 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:53.777388 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:53.849168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ff7f7e-c0fa-40e6-9794-e33a65d88c10 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:53.849168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ff7f7e-c0fa-40e6-9794-e33a65d88c10 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:53.849168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ff7f7e-c0fa-40e6-9794-e33a65d88c10 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:53.849168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 57/104] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:32:53 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:32:53.856234 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21eff308-46ae-4fa3-8b51-63b7d6111da4 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:53.856527 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21eff308-46ae-4fa3-8b51-63b7d6111da4 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:53.856807 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21eff308-46ae-4fa3-8b51-63b7d6111da4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:53.888790 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-21eff308-46ae-4fa3-8b51-63b7d6111da4 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:32:53.890007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-21eff308-46ae-4fa3-8b51-63b7d6111da4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:32:53.905305 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49fdb441-e80b-4e6b-bbd7-70b5b588a974 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:53.951381 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447da6bb-28a1-4fcf-b9a5-9b82eeb69031 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:53.952320 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447da6bb-28a1-4fcf-b9a5-9b82eeb69031 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:53.952609 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447da6bb-28a1-4fcf-b9a5-9b82eeb69031 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:53.952914 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447da6bb-28a1-4fcf-b9a5-9b82eeb69031 None admin] PATH_INFO: `/v3/roles/member` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:53.967470 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447da6bb-28a1-4fcf-b9a5-9b82eeb69031 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:32:53.976528 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-447da6bb-28a1-4fcf-b9a5-9b82eeb69031 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:32:53.976528 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:32:53.976528 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:32:53.976528 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:32:53.976528 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:32:53.976528 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:32:53.976528 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:32:53.976528 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:32:53.976528 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:32:53.976528 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:32:53.976528 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:32:53.976528 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:32:53.976528 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:32:53.976528 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:32:53.976528 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:32:53.976528 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:32:53.977503 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:32:53.977503 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:32:53.977503 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:32:53.977503 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:32:53.977503 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:32:53.977503 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:32:53.977503 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:32:53.977503 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:32:53.977503 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:32:53.977503 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:32:53.977503 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:32:53.977503 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:32:53.977503 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:32:53.977503 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:32:53.977503 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:32:53.977503 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:32:53.978423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:32:53.978423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:32:53.978423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:32:53.978423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:32:53.978423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:32:53.978423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:32:53.978423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:32:53.978423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:32:53.978423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:32:53.978423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:32:53.978423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:32:53.978423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:32:53.978423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 48/105] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:32:53 2022] GET /identity/v3/roles/member => generated 84 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:32:53.979576 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447da6bb-28a1-4fcf-b9a5-9b82eeb69031 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:53.997529 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 58/106] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:32:53 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 15:32:54.001661 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af37e8ff-c931-42ba-8e39-67acce5c00cf None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:54.002550 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af37e8ff-c931-42ba-8e39-67acce5c00cf None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:54.002550 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af37e8ff-c931-42ba-8e39-67acce5c00cf None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:54.003585 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 59/107] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:32:53 2022] GET /identity => generated 271 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:32:54.006903 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af37e8ff-c931-42ba-8e39-67acce5c00cf None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:54.136841 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:54.156415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447da6bb-28a1-4fcf-b9a5-9b82eeb69031 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:54.180023 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af37e8ff-c931-42ba-8e39-67acce5c00cf None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:54.206676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc3b0ee-0d88-4014-adfb-2b9771cd122f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:54.208180 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc3b0ee-0d88-4014-adfb-2b9771cd122f None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:54.208478 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc3b0ee-0d88-4014-adfb-2b9771cd122f None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:54.208752 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc3b0ee-0d88-4014-adfb-2b9771cd122f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:54.214986 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:54.224749 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc3b0ee-0d88-4014-adfb-2b9771cd122f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:32:54.227287 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dba2fe2-062e-4744-9cdb-c1bd2f8bd14f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:54.228229 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dba2fe2-062e-4744-9cdb-c1bd2f8bd14f None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:54.228708 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dba2fe2-062e-4744-9cdb-c1bd2f8bd14f None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:54.229173 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dba2fe2-062e-4744-9cdb-c1bd2f8bd14f None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:54.229517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-4cc3b0ee-0d88-4014-adfb-2b9771cd122f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:32:54.229517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:32:54.229517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:32:54.229517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:32:54.229517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:32:54.229517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:32:54.229517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:32:54.229517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:32:54.229517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:32:54.229517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:32:54.229517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:32:54.229517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:32:54.229517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:32:54.229517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:32:54.229517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:32:54.229517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:32:54.230312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:32:54.230312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:32:54.230312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:32:54.230312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:32:54.230312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:32:54.230312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:32:54.230312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:32:54.230312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:32:54.230312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:32:54.230312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:32:54.230312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:32:54.230312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:32:54.230312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:32:54.230312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:32:54.230312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:32:54.230312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:32:54.231125 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:32:54.231125 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:32:54.231125 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:32:54.231125 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:32:54.231125 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:32:54.231125 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:32:54.231125 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:32:54.231125 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:32:54.231125 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:32:54.231125 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:32:54.231125 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:32:54.231125 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:32:54.231125 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 49/108] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:32:53 2022] GET /identity/v3/roles/admin => generated 83 bytes in 252 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:32:54.231792 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06cf2bc-ecb6-4626-8c45-f73726a2e367 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:54.232608 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06cf2bc-ecb6-4626-8c45-f73726a2e367 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:54.232608 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06cf2bc-ecb6-4626-8c45-f73726a2e367 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:54.237566 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dba2fe2-062e-4744-9cdb-c1bd2f8bd14f None admin] RBAC: Adding query filter params (name=member) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:32:54.237566 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dba2fe2-062e-4744-9cdb-c1bd2f8bd14f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:54.238667 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dba2fe2-062e-4744-9cdb-c1bd2f8bd14f None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:54.249030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 60/109] 10.222.0.102 () {56 vars in 1106 bytes} [Wed Aug 24 15:32:54 2022] GET /identity/v3/roles?name=member => generated 343 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:54.250297 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dba2fe2-062e-4744-9cdb-c1bd2f8bd14f None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:54.256836 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-d06cf2bc-ecb6-4626-8c45-f73726a2e367 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:32:54.258397 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-d06cf2bc-ecb6-4626-8c45-f73726a2e367 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:32:54.339832 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 50/110] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:32:54 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:32:54.339832 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06cf2bc-ecb6-4626-8c45-f73726a2e367 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:54.386634 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dba2fe2-062e-4744-9cdb-c1bd2f8bd14f None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:54.441953 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcb55d6-2f09-4fbb-b7dc-23897ca36944 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:54.442616 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcb55d6-2f09-4fbb-b7dc-23897ca36944 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:54.442905 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcb55d6-2f09-4fbb-b7dc-23897ca36944 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:54.443189 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcb55d6-2f09-4fbb-b7dc-23897ca36944 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:54.446515 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:54.456277 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fcb55d6-2f09-4fbb-b7dc-23897ca36944 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:32:54.462087 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-6fcb55d6-2f09-4fbb-b7dc-23897ca36944 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 15:32:54.462087 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:32:54.462087 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:32:54.462087 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:32:54.462087 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:32:54.462087 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:32:54.462087 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:32:54.462087 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:32:54.462087 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:32:54.462087 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:32:54.462087 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:32:54.462087 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:32:54.462087 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:32:54.462087 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:32:54.462087 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:32:54.462087 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:32:54.463418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:32:54.463418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:32:54.463418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:32:54.463418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:32:54.463418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:32:54.463418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:32:54.463418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:32:54.463418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:32:54.463418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:32:54.463418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:32:54.463418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:32:54.463418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:32:54.463418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:32:54.463418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:32:54.463418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:32:54.463418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:32:54.465664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:32:54.465664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:32:54.465664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:32:54.465664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:32:54.465664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:32:54.465664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:32:54.465664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:32:54.465664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:32:54.465664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:32:54.465664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:32:54.465664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 15:32:54.465664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:32:54.465664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 61/111] 10.222.0.102 () {56 vars in 1119 bytes} [Wed Aug 24 15:32:54 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:32:54.465664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcb55d6-2f09-4fbb-b7dc-23897ca36944 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:54.489487 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06cf2bc-ecb6-4626-8c45-f73726a2e367 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:54.526409 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:54.534627 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da51965a-8117-4d89-9763-31e25066a730 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:54.535228 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da51965a-8117-4d89-9763-31e25066a730 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:54.535684 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da51965a-8117-4d89-9763-31e25066a730 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:54.535824 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da51965a-8117-4d89-9763-31e25066a730 None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:54.536498 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da51965a-8117-4d89-9763-31e25066a730 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:32:54.536806 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da51965a-8117-4d89-9763-31e25066a730 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:54.538202 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da51965a-8117-4d89-9763-31e25066a730 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:54.547113 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 51/112] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:32:54 2022] GET /identity/v3/roles?name=admin => generated 341 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:54.548180 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da51965a-8117-4d89-9763-31e25066a730 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:54.590775 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcb55d6-2f09-4fbb-b7dc-23897ca36944 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:54.630914 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860498c5-f604-4d55-9d09-786082866cbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:54.631695 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860498c5-f604-4d55-9d09-786082866cbb None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:54.632055 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860498c5-f604-4d55-9d09-786082866cbb None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:54.632410 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860498c5-f604-4d55-9d09-786082866cbb None admin] PATH_INFO: `/v3/users/cb1c2ffc97364fa48e0d109484b7871c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:54.633700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860498c5-f604-4d55-9d09-786082866cbb None admin] RBAC: Authorizing `identity:get_user(user_id=cb1c2ffc97364fa48e0d109484b7871c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:54.635319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860498c5-f604-4d55-9d09-786082866cbb None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:54.637057 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 62/113] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:32:54 2022] GET /identity/v3/users/cb1c2ffc97364fa48e0d109484b7871c => generated 276 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:54.641041 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860498c5-f604-4d55-9d09-786082866cbb None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:54.659988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da51965a-8117-4d89-9763-31e25066a730 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:54.698677 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e27396-942e-42f4-b8fa-8b7eb9264883 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:54.699316 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e27396-942e-42f4-b8fa-8b7eb9264883 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:54.699607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e27396-942e-42f4-b8fa-8b7eb9264883 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:54.699914 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e27396-942e-42f4-b8fa-8b7eb9264883 None admin] PATH_INFO: `/v3/domains/default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:54.701069 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e27396-942e-42f4-b8fa-8b7eb9264883 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:54.702526 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e27396-942e-42f4-b8fa-8b7eb9264883 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:54.707668 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 52/114] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:32:54 2022] GET /identity/v3/domains/default => generated 200 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:54.707668 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6e1ea8-6e5d-4685-8ae6-15a69dd9e1b7 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:54.707668 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6e1ea8-6e5d-4685-8ae6-15a69dd9e1b7 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:54.707668 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6e1ea8-6e5d-4685-8ae6-15a69dd9e1b7 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:54.712107 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 53/115] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:32:54 2022] GET /identity => generated 271 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:32:54.712794 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba6e1ea8-6e5d-4685-8ae6-15a69dd9e1b7 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:54.716978 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:54.753038 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860498c5-f604-4d55-9d09-786082866cbb None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:54.771655 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:54.794811 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a548851-7fbe-48c3-a91e-d631457602a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:54.795431 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a548851-7fbe-48c3-a91e-d631457602a9 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:54.795710 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a548851-7fbe-48c3-a91e-d631457602a9 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:54.795994 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a548851-7fbe-48c3-a91e-d631457602a9 None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:54.796714 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a548851-7fbe-48c3-a91e-d631457602a9 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:32:54.797026 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a548851-7fbe-48c3-a91e-d631457602a9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:54.799115 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a548851-7fbe-48c3-a91e-d631457602a9 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:54.808083 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 63/116] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:32:54 2022] GET /identity/v3/roles?name=anotherrole => generated 336 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:54.808930 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a548851-7fbe-48c3-a91e-d631457602a9 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:54.830493 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba6e1ea8-6e5d-4685-8ae6-15a69dd9e1b7 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:54.866871 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715b1cc8-361f-47a8-a8f8-ff09018fe79a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:54.867512 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715b1cc8-361f-47a8-a8f8-ff09018fe79a None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:54.867832 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715b1cc8-361f-47a8-a8f8-ff09018fe79a None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:54.868116 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715b1cc8-361f-47a8-a8f8-ff09018fe79a None admin] PATH_INFO: `/v3/users/3eb7904f17244b85baf19c438c53917f` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:54.869261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715b1cc8-361f-47a8-a8f8-ff09018fe79a None admin] RBAC: Authorizing `identity:get_user(user_id=3eb7904f17244b85baf19c438c53917f)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:54.870726 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715b1cc8-361f-47a8-a8f8-ff09018fe79a None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:54.872360 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 54/117] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:32:54 2022] GET /identity/v3/users/3eb7904f17244b85baf19c438c53917f => generated 248 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:54.877501 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715b1cc8-361f-47a8-a8f8-ff09018fe79a None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:54.917086 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a548851-7fbe-48c3-a91e-d631457602a9 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:54.953244 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebec32a6-a341-4acf-9bc3-92e7f968ce59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:54.954491 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebec32a6-a341-4acf-9bc3-92e7f968ce59 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:54.954769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebec32a6-a341-4acf-9bc3-92e7f968ce59 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:54.955095 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebec32a6-a341-4acf-9bc3-92e7f968ce59 None admin] PATH_INFO: `/v3/projects/749c6d651fc445cdb75db25a73873386` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:54.956327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebec32a6-a341-4acf-9bc3-92e7f968ce59 None admin] RBAC: Authorizing `identity:get_project(project_id=749c6d651fc445cdb75db25a73873386)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:54.957798 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebec32a6-a341-4acf-9bc3-92e7f968ce59 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:54.958391 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:54.959507 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 64/118] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:32:54 2022] GET /identity/v3/projects/749c6d651fc445cdb75db25a73873386 => generated 299 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:54.960372 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebec32a6-a341-4acf-9bc3-92e7f968ce59 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:54.991657 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715b1cc8-361f-47a8-a8f8-ff09018fe79a None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:55.014666 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:55.030239 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ebf96f-5187-4403-9112-33030eb08929 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:55.030880 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ebf96f-5187-4403-9112-33030eb08929 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:55.031165 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ebf96f-5187-4403-9112-33030eb08929 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:55.031447 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ebf96f-5187-4403-9112-33030eb08929 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:55.042517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ebf96f-5187-4403-9112-33030eb08929 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:32:55.047984 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-c8ebf96f-5187-4403-9112-33030eb08929 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 15:32:55.047984 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:32:55.047984 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:32:55.047984 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:32:55.047984 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:32:55.047984 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:32:55.047984 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:32:55.047984 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:32:55.047984 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:32:55.047984 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:32:55.047984 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:32:55.047984 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:32:55.047984 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:32:55.047984 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:32:55.047984 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:32:55.047984 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:32:55.049155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:32:55.049155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:32:55.049155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:32:55.049155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:32:55.049155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:32:55.049155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:32:55.049155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:32:55.049155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:32:55.049155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:32:55.049155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:32:55.049155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:32:55.049155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:32:55.049155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:32:55.049155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:32:55.049155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:32:55.049155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:32:55.050202 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:32:55.050202 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:32:55.050202 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:32:55.050202 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:32:55.050202 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:32:55.050202 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:32:55.050202 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:32:55.050202 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:32:55.050202 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:32:55.050202 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:32:55.050202 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 15:32:55.050202 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:32:55.050202 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 55/119] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:32:54 2022] GET /identity/v3/roles/reader => generated 84 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:32:55.051177 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a777eb-d29b-44d8-8409-06f7b88d9ea4 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:55.051177 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a777eb-d29b-44d8-8409-06f7b88d9ea4 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:55.051326 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a777eb-d29b-44d8-8409-06f7b88d9ea4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:55.070052 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-63a777eb-d29b-44d8-8409-06f7b88d9ea4 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:32:55.071269 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-63a777eb-d29b-44d8-8409-06f7b88d9ea4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:32:55.078254 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebec32a6-a341-4acf-9bc3-92e7f968ce59 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:55.112632 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163e39d5-b601-48ff-8343-f51682b72e88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:55.113242 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163e39d5-b601-48ff-8343-f51682b72e88 None admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:55.113527 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163e39d5-b601-48ff-8343-f51682b72e88 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:55.113805 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163e39d5-b601-48ff-8343-f51682b72e88 None admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:55.114455 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-163e39d5-b601-48ff-8343-f51682b72e88 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:55.115962 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-163e39d5-b601-48ff-8343-f51682b72e88 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:55.144522 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 56/120] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:32:55 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:32:55.153991 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 65/121] 10.222.0.102 () {60 vars in 1146 bytes} [Wed Aug 24 15:32:54 2022] POST /identity/v3/projects => generated 303 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:32:55.153991 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163e39d5-b601-48ff-8343-f51682b72e88 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:55.154265 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a777eb-d29b-44d8-8409-06f7b88d9ea4 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:55.234427 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:55.292545 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a777eb-d29b-44d8-8409-06f7b88d9ea4 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:55.293560 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:55.301276 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163e39d5-b601-48ff-8343-f51682b72e88 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:55.346266 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894720ba-d7cd-4eaa-a311-8dc2edb5038a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:55.346765 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a995fe63-ecef-48c0-a8ad-e1ef41844582 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:55.346907 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894720ba-d7cd-4eaa-a311-8dc2edb5038a None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:55.347192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894720ba-d7cd-4eaa-a311-8dc2edb5038a None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:55.347474 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894720ba-d7cd-4eaa-a311-8dc2edb5038a None admin] PATH_INFO: `/v3/projects/749c6d651fc445cdb75db25a73873386` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:55.348245 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a995fe63-ecef-48c0-a8ad-e1ef41844582 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:55.348676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-894720ba-d7cd-4eaa-a311-8dc2edb5038a None admin] RBAC: Authorizing `identity:get_project(project_id=749c6d651fc445cdb75db25a73873386)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:55.348940 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a995fe63-ecef-48c0-a8ad-e1ef41844582 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:55.349418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a995fe63-ecef-48c0-a8ad-e1ef41844582 None admin] PATH_INFO: `/v3/users/cb1c2ffc97364fa48e0d109484b7871c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:55.350189 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-894720ba-d7cd-4eaa-a311-8dc2edb5038a None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:55.350988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a995fe63-ecef-48c0-a8ad-e1ef41844582 None admin] RBAC: Authorizing `identity:get_user(user_id=cb1c2ffc97364fa48e0d109484b7871c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:55.352247 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 66/122] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:32:55 2022] GET /identity/v3/projects/749c6d651fc445cdb75db25a73873386 => generated 299 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:55.352982 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a995fe63-ecef-48c0-a8ad-e1ef41844582 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:55.353177 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894720ba-d7cd-4eaa-a311-8dc2edb5038a None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:55.355242 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 57/123] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:32:55 2022] GET /identity/v3/users/cb1c2ffc97364fa48e0d109484b7871c => generated 276 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:55.359021 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a995fe63-ecef-48c0-a8ad-e1ef41844582 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:55.481327 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a995fe63-ecef-48c0-a8ad-e1ef41844582 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:55.482180 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894720ba-d7cd-4eaa-a311-8dc2edb5038a None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:55.511526 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:55.518693 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2bd764-dddd-43a8-916c-8ba14f2d3ddb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:55.519305 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2bd764-dddd-43a8-916c-8ba14f2d3ddb None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:55.519588 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2bd764-dddd-43a8-916c-8ba14f2d3ddb None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:55.520122 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2bd764-dddd-43a8-916c-8ba14f2d3ddb None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:55.520662 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2bd764-dddd-43a8-916c-8ba14f2d3ddb None admin] RBAC: Adding query filter params (name=reader) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:32:55.520868 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2bd764-dddd-43a8-916c-8ba14f2d3ddb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:55.522288 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2bd764-dddd-43a8-916c-8ba14f2d3ddb None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:55.525518 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eac6bb7-877f-416c-bcdd-38e4d0dec385 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:55.526142 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eac6bb7-877f-416c-bcdd-38e4d0dec385 None admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:55.526429 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eac6bb7-877f-416c-bcdd-38e4d0dec385 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:55.526711 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eac6bb7-877f-416c-bcdd-38e4d0dec385 None admin] PATH_INFO: `/v3/projects/749c6d651fc445cdb75db25a73873386/users/cb1c2ffc97364fa48e0d109484b7871c/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:55.528800 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eac6bb7-877f-416c-bcdd-38e4d0dec385 None admin] RBAC: Authorizing `identity:create_grant(project_id=749c6d651fc445cdb75db25a73873386, user_id=cb1c2ffc97364fa48e0d109484b7871c, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:55.530282 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eac6bb7-877f-416c-bcdd-38e4d0dec385 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:55.531298 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 58/124] 10.222.0.102 () {56 vars in 1106 bytes} [Wed Aug 24 15:32:55 2022] GET /identity/v3/roles?name=reader => generated 343 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:55.533743 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2bd764-dddd-43a8-916c-8ba14f2d3ddb None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:55.545183 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 67/125] 10.222.0.102 () {58 vars in 1444 bytes} [Wed Aug 24 15:32:55 2022] PUT /identity/v3/projects/749c6d651fc445cdb75db25a73873386/users/cb1c2ffc97364fa48e0d109484b7871c/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:32:55.546023 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eac6bb7-877f-416c-bcdd-38e4d0dec385 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:55.590466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:55.681973 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2bd764-dddd-43a8-916c-8ba14f2d3ddb None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:55.695824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eac6bb7-877f-416c-bcdd-38e4d0dec385 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:55.732224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21833405-1569-4adf-b4df-b016829ce78a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:55.732896 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21833405-1569-4adf-b4df-b016829ce78a None admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:55.733179 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21833405-1569-4adf-b4df-b016829ce78a None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:55.733456 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21833405-1569-4adf-b4df-b016829ce78a None admin] PATH_INFO: `/v3/projects/749c6d651fc445cdb75db25a73873386/users/3eb7904f17244b85baf19c438c53917f/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:55.735463 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21833405-1569-4adf-b4df-b016829ce78a None admin] RBAC: Authorizing `identity:create_grant(project_id=749c6d651fc445cdb75db25a73873386, user_id=3eb7904f17244b85baf19c438c53917f, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:55.736969 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21833405-1569-4adf-b4df-b016829ce78a None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:55.745509 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628d5b7f-f43a-44b9-906c-aa76db493f38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:55.747230 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628d5b7f-f43a-44b9-906c-aa76db493f38 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:55.747774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628d5b7f-f43a-44b9-906c-aa76db493f38 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:55.748112 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628d5b7f-f43a-44b9-906c-aa76db493f38 None admin] PATH_INFO: `/v3/projects/749c6d651fc445cdb75db25a73873386` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:55.750063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-628d5b7f-f43a-44b9-906c-aa76db493f38 None admin] RBAC: Authorizing `identity:get_project(project_id=749c6d651fc445cdb75db25a73873386)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:55.752484 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-628d5b7f-f43a-44b9-906c-aa76db493f38 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:55.754772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 68/126] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:32:55 2022] GET /identity/v3/projects/749c6d651fc445cdb75db25a73873386 => generated 299 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:55.756818 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628d5b7f-f43a-44b9-906c-aa76db493f38 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:55.763922 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 59/127] 10.222.0.102 () {58 vars in 1444 bytes} [Wed Aug 24 15:32:55 2022] PUT /identity/v3/projects/749c6d651fc445cdb75db25a73873386/users/3eb7904f17244b85baf19c438c53917f/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:32:55.764868 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21833405-1569-4adf-b4df-b016829ce78a None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:55.866303 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:55.867998 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:55.874396 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628d5b7f-f43a-44b9-906c-aa76db493f38 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:55.914305 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a27e7a5-272c-4791-be16-b9fcdc0c25d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:55.915465 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a27e7a5-272c-4791-be16-b9fcdc0c25d0 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:55.916050 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a27e7a5-272c-4791-be16-b9fcdc0c25d0 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:55.916564 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a27e7a5-272c-4791-be16-b9fcdc0c25d0 None admin] PATH_INFO: `/v3/roles/member` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:55.930275 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a27e7a5-272c-4791-be16-b9fcdc0c25d0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:32:55.935657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-4a27e7a5-272c-4791-be16-b9fcdc0c25d0 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:32:55.935657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:32:55.935657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:32:55.935657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:32:55.935657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:32:55.935657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:32:55.935657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:32:55.935657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:32:55.935657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:32:55.935657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:32:55.935657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:32:55.935657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:32:55.935657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:32:55.935657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:32:55.935657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:32:55.935657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:32:55.936579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:32:55.936579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:32:55.936579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:32:55.936579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:32:55.936579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:32:55.936579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:32:55.936579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:32:55.936579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:32:55.936579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:32:55.936579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:32:55.936579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:32:55.936579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:32:55.936579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:32:55.936579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:32:55.936579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:32:55.936579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:32:55.937345 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:32:55.937345 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:32:55.937345 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:32:55.937345 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:32:55.937345 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:32:55.937345 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:32:55.937345 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:32:55.937345 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:32:55.937345 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:32:55.937345 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:32:55.937345 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:32:55.937345 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:32:55.940987 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 69/128] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:32:55 2022] GET /identity/v3/roles/member => generated 84 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:32:55.942048 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a27e7a5-272c-4791-be16-b9fcdc0c25d0 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:55.990731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21833405-1569-4adf-b4df-b016829ce78a None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:56.051751 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d169f84c-d5e4-4b4d-b9f4-93ff45a3bc51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:56.052804 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d169f84c-d5e4-4b4d-b9f4-93ff45a3bc51 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:56.052804 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d169f84c-d5e4-4b4d-b9f4-93ff45a3bc51 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:56.052997 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d169f84c-d5e4-4b4d-b9f4-93ff45a3bc51 None admin] PATH_INFO: `/v3/users/ed4c3aa4c7b14a35b57177bf48a0e250` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:56.062935 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a27e7a5-272c-4791-be16-b9fcdc0c25d0 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:56.078131 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d169f84c-d5e4-4b4d-b9f4-93ff45a3bc51 None admin] RBAC: Authorizing `identity:get_user(user_id=ed4c3aa4c7b14a35b57177bf48a0e250)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:56.080344 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d169f84c-d5e4-4b4d-b9f4-93ff45a3bc51 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:56.081887 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 60/129] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:32:55 2022] GET /identity/v3/users/ed4c3aa4c7b14a35b57177bf48a0e250 => generated 290 bytes in 317 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:56.082674 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d169f84c-d5e4-4b4d-b9f4-93ff45a3bc51 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:56.102241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e273b377-5cdb-415b-b158-6fcc5176a6b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:56.103243 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e273b377-5cdb-415b-b158-6fcc5176a6b6 None admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:56.103766 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e273b377-5cdb-415b-b158-6fcc5176a6b6 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:56.104241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e273b377-5cdb-415b-b158-6fcc5176a6b6 None admin] PATH_INFO: `/v3/projects/749c6d651fc445cdb75db25a73873386/users/cb1c2ffc97364fa48e0d109484b7871c/roles/b8429ebd9c2a4cbbbbf30ad56ce23bfa` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:56.106639 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e273b377-5cdb-415b-b158-6fcc5176a6b6 None admin] RBAC: Authorizing `identity:create_grant(project_id=749c6d651fc445cdb75db25a73873386, user_id=cb1c2ffc97364fa48e0d109484b7871c, role_id=b8429ebd9c2a4cbbbbf30ad56ce23bfa)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:56.108339 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e273b377-5cdb-415b-b158-6fcc5176a6b6 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:56.128517 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 70/130] 10.222.0.102 () {58 vars in 1444 bytes} [Wed Aug 24 15:32:55 2022] PUT /identity/v3/projects/749c6d651fc445cdb75db25a73873386/users/cb1c2ffc97364fa48e0d109484b7871c/roles/b8429ebd9c2a4cbbbbf30ad56ce23bfa => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:32:56.130583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e273b377-5cdb-415b-b158-6fcc5176a6b6 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:56.161870 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:56.301906 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d169f84c-d5e4-4b4d-b9f4-93ff45a3bc51 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:56.320474 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:56.335131 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e273b377-5cdb-415b-b158-6fcc5176a6b6 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:56.384626 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f8086f-4726-49eb-b3d9-260b905d11e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:56.385250 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f8086f-4726-49eb-b3d9-260b905d11e7 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:56.385530 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f8086f-4726-49eb-b3d9-260b905d11e7 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:56.385802 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f8086f-4726-49eb-b3d9-260b905d11e7 None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:56.386484 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f8086f-4726-49eb-b3d9-260b905d11e7 None admin] RBAC: Adding query filter params (name=member) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:32:56.386788 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f8086f-4726-49eb-b3d9-260b905d11e7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:56.388255 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f8086f-4726-49eb-b3d9-260b905d11e7 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:56.403398 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 61/131] 10.222.0.102 () {56 vars in 1106 bytes} [Wed Aug 24 15:32:56 2022] GET /identity/v3/roles?name=member => generated 343 bytes in 321 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:56.413861 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f8086f-4726-49eb-b3d9-260b905d11e7 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:56.419787 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d268e3-ab9e-4258-ac5f-3b53b03b7647 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:56.420425 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d268e3-ab9e-4258-ac5f-3b53b03b7647 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:56.420706 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d268e3-ab9e-4258-ac5f-3b53b03b7647 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:56.420983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d268e3-ab9e-4258-ac5f-3b53b03b7647 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:56.537150 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21d268e3-ab9e-4258-ac5f-3b53b03b7647 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:56.538932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21d268e3-ab9e-4258-ac5f-3b53b03b7647 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:56.560569 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:56.633957 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f8086f-4726-49eb-b3d9-260b905d11e7 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:56.661247 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 71/132] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:32:56 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 532 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:32:56.667212 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d268e3-ab9e-4258-ac5f-3b53b03b7647 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:56.705793 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5724094a-f413-4757-bd69-26c01fc040aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:56.706452 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5724094a-f413-4757-bd69-26c01fc040aa None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:56.706742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5724094a-f413-4757-bd69-26c01fc040aa None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:56.707021 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5724094a-f413-4757-bd69-26c01fc040aa None admin] PATH_INFO: `/v3/users/cb1c2ffc97364fa48e0d109484b7871c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:56.712438 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5724094a-f413-4757-bd69-26c01fc040aa None admin] RBAC: Authorizing `identity:get_user(user_id=cb1c2ffc97364fa48e0d109484b7871c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:56.713963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5724094a-f413-4757-bd69-26c01fc040aa None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:56.715997 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 62/133] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:32:56 2022] GET /identity/v3/users/cb1c2ffc97364fa48e0d109484b7871c => generated 276 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:56.721543 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5724094a-f413-4757-bd69-26c01fc040aa None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:56.794607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:56.808296 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d268e3-ab9e-4258-ac5f-3b53b03b7647 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:56.827657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:56.872416 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbfd8e06-3ac1-44fe-b63c-7a435eb4b309 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:56.873093 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbfd8e06-3ac1-44fe-b63c-7a435eb4b309 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:56.873385 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbfd8e06-3ac1-44fe-b63c-7a435eb4b309 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:56.873670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbfd8e06-3ac1-44fe-b63c-7a435eb4b309 None admin] PATH_INFO: `/v3/projects/749c6d651fc445cdb75db25a73873386` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:56.874930 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbfd8e06-3ac1-44fe-b63c-7a435eb4b309 None admin] RBAC: Authorizing `identity:get_project(project_id=749c6d651fc445cdb75db25a73873386)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:56.877164 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbfd8e06-3ac1-44fe-b63c-7a435eb4b309 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:56.878879 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 72/134] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:32:56 2022] GET /identity/v3/projects/749c6d651fc445cdb75db25a73873386 => generated 299 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:56.884834 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbfd8e06-3ac1-44fe-b63c-7a435eb4b309 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:56.899351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5724094a-f413-4757-bd69-26c01fc040aa None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:56.936898 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244a7d35-86bd-45f0-82f1-958b00f116c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:56.937450 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244a7d35-86bd-45f0-82f1-958b00f116c5 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:56.938058 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244a7d35-86bd-45f0-82f1-958b00f116c5 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:56.938380 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244a7d35-86bd-45f0-82f1-958b00f116c5 None admin] PATH_INFO: `/v3/projects/2dcca3c9f7d64026b4e21bfca99e2ab8` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:56.939596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244a7d35-86bd-45f0-82f1-958b00f116c5 None admin] RBAC: Authorizing `identity:get_project(project_id=2dcca3c9f7d64026b4e21bfca99e2ab8)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:56.942489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244a7d35-86bd-45f0-82f1-958b00f116c5 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:56.944187 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 63/135] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:32:56 2022] GET /identity/v3/projects/2dcca3c9f7d64026b4e21bfca99e2ab8 => generated 313 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:32:56.953186 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244a7d35-86bd-45f0-82f1-958b00f116c5 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:57.088428 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244a7d35-86bd-45f0-82f1-958b00f116c5 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:57.098128 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbfd8e06-3ac1-44fe-b63c-7a435eb4b309 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:57.098622 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:57.126332 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449e4dfb-2664-4be8-98c8-b301be960d18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:57.127270 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449e4dfb-2664-4be8-98c8-b301be960d18 None admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:57.127718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449e4dfb-2664-4be8-98c8-b301be960d18 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:57.128008 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449e4dfb-2664-4be8-98c8-b301be960d18 None admin] PATH_INFO: `/v3/projects/2dcca3c9f7d64026b4e21bfca99e2ab8/users/cb1c2ffc97364fa48e0d109484b7871c/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:57.130628 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449e4dfb-2664-4be8-98c8-b301be960d18 None admin] RBAC: Authorizing `identity:create_grant(project_id=2dcca3c9f7d64026b4e21bfca99e2ab8, user_id=cb1c2ffc97364fa48e0d109484b7871c, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:57.132226 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449e4dfb-2664-4be8-98c8-b301be960d18 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:57.152484 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 64/136] 10.222.0.102 () {58 vars in 1444 bytes} [Wed Aug 24 15:32:56 2022] PUT /identity/v3/projects/2dcca3c9f7d64026b4e21bfca99e2ab8/users/cb1c2ffc97364fa48e0d109484b7871c/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:32:57.168321 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7950bb40-e841-4b59-b146-1d28a4fba138 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:57.169068 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7950bb40-e841-4b59-b146-1d28a4fba138 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:57.169427 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7950bb40-e841-4b59-b146-1d28a4fba138 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:57.169773 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7950bb40-e841-4b59-b146-1d28a4fba138 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:57.170654 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7950bb40-e841-4b59-b146-1d28a4fba138 None admin] RBAC: Adding query filter params (role.id=868817cab1414b5aa207bc5bdf4f1d9a, scope.project.id=749c6d651fc445cdb75db25a73873386, user.id=ed4c3aa4c7b14a35b57177bf48a0e250) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:32:57.171078 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7950bb40-e841-4b59-b146-1d28a4fba138 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:57.173565 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7950bb40-e841-4b59-b146-1d28a4fba138 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:57.180605 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 73/137] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:32:56 2022] GET /identity/v3/role_assignments?user.id=ed4c3aa4c7b14a35b57177bf48a0e250&scope.project.id=749c6d651fc445cdb75db25a73873386&role.id=868817cab1414b5aa207bc5bdf4f1d9a => generated 260 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:32:58.117013 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cad7f3d-53ba-4459-8d46-7c586900d746 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:58.117509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cad7f3d-53ba-4459-8d46-7c586900d746 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:58.117509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cad7f3d-53ba-4459-8d46-7c586900d746 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:58.119142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 65/138] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:32:58 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:32:58.124532 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e970e8-456e-4e4b-9fa1-5c5cd233f2f7 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:58.124976 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e970e8-456e-4e4b-9fa1-5c5cd233f2f7 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:58.125172 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e970e8-456e-4e4b-9fa1-5c5cd233f2f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:58.151222 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-f5e970e8-456e-4e4b-9fa1-5c5cd233f2f7 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:32:58.152824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-f5e970e8-456e-4e4b-9fa1-5c5cd233f2f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:32:58.230205 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:58.278631 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 74/139] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:32:58 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:32:58.510789 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d3d73c-718e-4c38-84e3-8aac8c467806 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:58.511486 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d3d73c-718e-4c38-84e3-8aac8c467806 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:58.511982 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d3d73c-718e-4c38-84e3-8aac8c467806 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:58.513520 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 66/140] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:32:58 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:32:58.519867 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b49994-5d2a-40da-b903-4e72dc59f44b None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:58.520161 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b49994-5d2a-40da-b903-4e72dc59f44b None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:58.520448 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b49994-5d2a-40da-b903-4e72dc59f44b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:58.548808 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eead5ff3-96bc-4cd4-9b30-71cf3155bc76 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:58.549722 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eead5ff3-96bc-4cd4-9b30-71cf3155bc76 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:58.549868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-88b49994-5d2a-40da-b903-4e72dc59f44b None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:32:58.550649 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eead5ff3-96bc-4cd4-9b30-71cf3155bc76 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:58.551562 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-88b49994-5d2a-40da-b903-4e72dc59f44b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:32:58.553370 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 67/141] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:32:58 2022] GET /identity => generated 271 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:32:58.559282 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256867e8-d089-4104-be61-b325d28dbe78 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:58.560010 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256867e8-d089-4104-be61-b325d28dbe78 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:58.560507 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256867e8-d089-4104-be61-b325d28dbe78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:58.585609 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-256867e8-d089-4104-be61-b325d28dbe78 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:32:58.587440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-256867e8-d089-4104-be61-b325d28dbe78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:32:58.702234 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 75/142] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:32:58 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:32:58.793145 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 68/143] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:32:58 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:32:58.889589 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b49994-5d2a-40da-b903-4e72dc59f44b None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:59.034960 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b49994-5d2a-40da-b903-4e72dc59f44b None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:59.051904 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:59.090289 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143c60e7-035e-4c04-85b8-5cb6715591e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:59.091039 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143c60e7-035e-4c04-85b8-5cb6715591e8 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:59.091321 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143c60e7-035e-4c04-85b8-5cb6715591e8 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:59.091597 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143c60e7-035e-4c04-85b8-5cb6715591e8 None admin] PATH_INFO: `/v3/domains/default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:59.093226 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143c60e7-035e-4c04-85b8-5cb6715591e8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:59.096873 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143c60e7-035e-4c04-85b8-5cb6715591e8 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:59.097925 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 76/144] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:32:58 2022] GET /identity/v3/domains/default => generated 200 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:59.108803 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256867e8-d089-4104-be61-b325d28dbe78 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:59.141855 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143c60e7-035e-4c04-85b8-5cb6715591e8 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:59.287386 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143c60e7-035e-4c04-85b8-5cb6715591e8 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:59.343459 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:59.349896 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256867e8-d089-4104-be61-b325d28dbe78 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:59.363825 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d067ae-6c3b-4f7b-95f5-6c97086343d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:59.364758 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d067ae-6c3b-4f7b-95f5-6c97086343d9 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:59.365190 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d067ae-6c3b-4f7b-95f5-6c97086343d9 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:59.365601 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d067ae-6c3b-4f7b-95f5-6c97086343d9 None admin] PATH_INFO: `/v3/roles/member` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:59.378588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d067ae-6c3b-4f7b-95f5-6c97086343d9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:32:59.384558 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-77d067ae-6c3b-4f7b-95f5-6c97086343d9 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:32:59.384558 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:32:59.384558 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:32:59.384558 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:32:59.384558 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:32:59.384558 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:32:59.384558 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:32:59.384558 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:32:59.384558 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:32:59.384558 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:32:59.384558 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:32:59.384558 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:32:59.384558 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:32:59.384558 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:32:59.384558 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:32:59.384558 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:32:59.385462 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:32:59.385462 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:32:59.385462 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:32:59.385462 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:32:59.385462 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:32:59.385462 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:32:59.385462 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:32:59.385462 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:32:59.385462 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:32:59.385462 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:32:59.385462 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:32:59.385462 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:32:59.385462 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:32:59.385462 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:32:59.385462 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:32:59.385462 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:32:59.386215 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:32:59.386215 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:32:59.386215 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:32:59.386215 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:32:59.386215 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:32:59.386215 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:32:59.386215 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:32:59.386215 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:32:59.386215 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:32:59.386215 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:32:59.386215 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:32:59.386215 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:32:59.388625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 77/145] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:32:59 2022] GET /identity/v3/roles/member => generated 84 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:32:59.390272 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff69a2a-154e-4a2c-8b9a-8346ef1a3e91 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:59.391020 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff69a2a-154e-4a2c-8b9a-8346ef1a3e91 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:59.392267 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff69a2a-154e-4a2c-8b9a-8346ef1a3e91 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:59.394585 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 78/146] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:32:59 2022] GET /identity => generated 271 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:32:59.395597 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff69a2a-154e-4a2c-8b9a-8346ef1a3e91 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:59.408380 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a082094c-894e-4352-a44b-c713da656bf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:59.409131 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a082094c-894e-4352-a44b-c713da656bf6 None admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:59.409528 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a082094c-894e-4352-a44b-c713da656bf6 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:59.410297 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a082094c-894e-4352-a44b-c713da656bf6 None admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:59.410990 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a082094c-894e-4352-a44b-c713da656bf6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:59.412926 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a082094c-894e-4352-a44b-c713da656bf6 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:59.442187 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:59.444836 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 69/147] 10.222.0.102 () {60 vars in 1137 bytes} [Wed Aug 24 15:32:59 2022] POST /identity/v3/users => generated 284 bytes in 337 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:32:59.452669 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a082094c-894e-4352-a44b-c713da656bf6 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:59.597831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff69a2a-154e-4a2c-8b9a-8346ef1a3e91 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:59.682361 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a082094c-894e-4352-a44b-c713da656bf6 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:59.682546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5798d4c2-73c3-465d-9235-dc8fd8634144 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:59.683175 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5798d4c2-73c3-465d-9235-dc8fd8634144 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:59.683475 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5798d4c2-73c3-465d-9235-dc8fd8634144 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:59.683767 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5798d4c2-73c3-465d-9235-dc8fd8634144 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:59.712656 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5798d4c2-73c3-465d-9235-dc8fd8634144 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:32:59.722639 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-5798d4c2-73c3-465d-9235-dc8fd8634144 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:32:59.722639 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:32:59.722639 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:32:59.722639 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:32:59.722639 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:32:59.722639 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:32:59.722639 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:32:59.722639 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:32:59.722639 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:32:59.722639 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:32:59.722639 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:32:59.722639 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:32:59.722639 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:32:59.722639 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:32:59.722639 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:32:59.722639 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:32:59.723546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:32:59.723546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:32:59.723546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:32:59.723546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:32:59.723546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:32:59.723546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:32:59.723546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:32:59.723546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:32:59.723546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:32:59.723546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:32:59.723546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:32:59.723546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:32:59.723546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:32:59.723546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:32:59.723546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:32:59.723546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:32:59.724368 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:32:59.724368 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:32:59.724368 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:32:59.724368 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:32:59.724368 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:32:59.724368 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:32:59.724368 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:32:59.724368 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:32:59.724368 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:32:59.724368 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:32:59.724368 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:32:59.724368 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:32:59.724368 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 79/148] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:32:59 2022] GET /identity/v3/roles/admin => generated 83 bytes in 329 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:32:59.725747 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e8b9b2-01cc-4083-bd74-222a3629cfa9 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:59.727405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e8b9b2-01cc-4083-bd74-222a3629cfa9 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:59.727753 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e8b9b2-01cc-4083-bd74-222a3629cfa9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:59.751431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d14137-867c-4449-9452-2ba5c08e0746 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:59.752095 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d14137-867c-4449-9452-2ba5c08e0746 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:59.752383 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d14137-867c-4449-9452-2ba5c08e0746 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:59.752809 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d14137-867c-4449-9452-2ba5c08e0746 None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:59.753507 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d14137-867c-4449-9452-2ba5c08e0746 None admin] RBAC: Adding query filter params (name=member) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:32:59.753677 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-85e8b9b2-01cc-4083-bd74-222a3629cfa9 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:32:59.753828 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d14137-867c-4449-9452-2ba5c08e0746 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:59.754879 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-85e8b9b2-01cc-4083-bd74-222a3629cfa9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:32:59.755340 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d14137-867c-4449-9452-2ba5c08e0746 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:32:59.769517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 70/149] 10.222.0.102 () {56 vars in 1106 bytes} [Wed Aug 24 15:32:59 2022] GET /identity/v3/roles?name=member => generated 343 bytes in 324 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:32:59.770617 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d14137-867c-4449-9452-2ba5c08e0746 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:59.800069 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:59.884568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 80/150] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:32:59 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:32:59.885448 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e8b9b2-01cc-4083-bd74-222a3629cfa9 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:32:59.920089 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:32:59.930021 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d14137-867c-4449-9452-2ba5c08e0746 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:32:59.992837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf0ae4e-0166-4978-9663-dc35e3b19d53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:32:59.993827 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf0ae4e-0166-4978-9663-dc35e3b19d53 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:32:59.994319 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf0ae4e-0166-4978-9663-dc35e3b19d53 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:32:59.994854 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf0ae4e-0166-4978-9663-dc35e3b19d53 None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:32:59.995899 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf0ae4e-0166-4978-9663-dc35e3b19d53 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:32:59.996407 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf0ae4e-0166-4978-9663-dc35e3b19d53 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:32:59.998193 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf0ae4e-0166-4978-9663-dc35e3b19d53 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:00.017474 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 71/151] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:32:59 2022] GET /identity/v3/roles?name=admin => generated 341 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:00.022075 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e8b9b2-01cc-4083-bd74-222a3629cfa9 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:00.022972 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf0ae4e-0166-4978-9663-dc35e3b19d53 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:00.066300 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c605867b-263f-4090-8461-39b3f11ad980 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:00.067295 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c605867b-263f-4090-8461-39b3f11ad980 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:00.067583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c605867b-263f-4090-8461-39b3f11ad980 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:00.067885 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c605867b-263f-4090-8461-39b3f11ad980 None admin] PATH_INFO: `/v3/users/cb1c2ffc97364fa48e0d109484b7871c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:00.069152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c605867b-263f-4090-8461-39b3f11ad980 None admin] RBAC: Authorizing `identity:get_user(user_id=cb1c2ffc97364fa48e0d109484b7871c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:00.070759 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c605867b-263f-4090-8461-39b3f11ad980 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:00.072571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 81/152] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:32:59 2022] GET /identity/v3/users/cb1c2ffc97364fa48e0d109484b7871c => generated 276 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:00.073766 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf981950-1627-444e-8a51-547d67a76e09 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:00.074047 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf981950-1627-444e-8a51-547d67a76e09 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:00.074373 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf981950-1627-444e-8a51-547d67a76e09 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:00.084609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 82/153] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:00 2022] GET /identity => generated 271 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:00.084609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf981950-1627-444e-8a51-547d67a76e09 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:00.127701 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:00.153345 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf0ae4e-0166-4978-9663-dc35e3b19d53 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:00.194545 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f69fc6-475b-4c39-92e9-1db625325eb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:00.195196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f69fc6-475b-4c39-92e9-1db625325eb4 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:00.195477 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f69fc6-475b-4c39-92e9-1db625325eb4 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:00.195787 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f69fc6-475b-4c39-92e9-1db625325eb4 None admin] PATH_INFO: `/v3/users/3eb7904f17244b85baf19c438c53917f` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:00.198006 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f69fc6-475b-4c39-92e9-1db625325eb4 None admin] RBAC: Authorizing `identity:get_user(user_id=3eb7904f17244b85baf19c438c53917f)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:00.199234 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:00.200323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f69fc6-475b-4c39-92e9-1db625325eb4 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:00.202371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 72/154] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:33:00 2022] GET /identity/v3/users/3eb7904f17244b85baf19c438c53917f => generated 248 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:33:00.204555 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47691bfa-87b9-42ba-a54a-518a660e90bd None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:00.204831 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47691bfa-87b9-42ba-a54a-518a660e90bd None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:00.205107 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47691bfa-87b9-42ba-a54a-518a660e90bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:00.219732 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf981950-1627-444e-8a51-547d67a76e09 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:00.228508 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-47691bfa-87b9-42ba-a54a-518a660e90bd None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:00.229748 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-47691bfa-87b9-42ba-a54a-518a660e90bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:00.288833 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27266dbe-2d34-484b-8c18-fb10d8394d6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:00.290160 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27266dbe-2d34-484b-8c18-fb10d8394d6e None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:00.290953 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27266dbe-2d34-484b-8c18-fb10d8394d6e None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:00.291767 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27266dbe-2d34-484b-8c18-fb10d8394d6e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:00.315493 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 73/155] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:00 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:00.316487 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47691bfa-87b9-42ba-a54a-518a660e90bd None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:00.408743 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:00.411900 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27266dbe-2d34-484b-8c18-fb10d8394d6e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:00.413873 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27266dbe-2d34-484b-8c18-fb10d8394d6e None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:00.435687 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47691bfa-87b9-42ba-a54a-518a660e90bd None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:00.479507 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13a461d-169d-45c2-a40f-523e2c41f21e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:00.480185 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13a461d-169d-45c2-a40f-523e2c41f21e None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:00.480466 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13a461d-169d-45c2-a40f-523e2c41f21e None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:00.480794 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13a461d-169d-45c2-a40f-523e2c41f21e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:00.526475 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:00.539658 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 83/156] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:33:00 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 458 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 24 15:33:00.540479 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27266dbe-2d34-484b-8c18-fb10d8394d6e None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:00.561822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f13a461d-169d-45c2-a40f-523e2c41f21e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:00.564260 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f13a461d-169d-45c2-a40f-523e2c41f21e None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:00.652497 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:00.664131 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27266dbe-2d34-484b-8c18-fb10d8394d6e None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:00.686589 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 74/157] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:33:00 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:33:00.689248 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f026f4-5aac-443b-8c38-e5c1eca0e27b None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:00.689596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f026f4-5aac-443b-8c38-e5c1eca0e27b None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:00.689877 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f026f4-5aac-443b-8c38-e5c1eca0e27b None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:00.691353 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 75/158] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:00 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:00.692193 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f026f4-5aac-443b-8c38-e5c1eca0e27b None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:00.723018 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db5194c-2f8d-4115-84f0-06997f73b3eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:00.724763 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db5194c-2f8d-4115-84f0-06997f73b3eb None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:00.725538 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db5194c-2f8d-4115-84f0-06997f73b3eb None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:00.726450 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db5194c-2f8d-4115-84f0-06997f73b3eb None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:00.750664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db5194c-2f8d-4115-84f0-06997f73b3eb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:00.757899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-4db5194c-2f8d-4115-84f0-06997f73b3eb None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 15:33:00.757899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:00.757899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:00.757899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:00.757899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:00.757899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:00.757899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:00.757899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:00.757899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:00.757899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:00.757899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:00.757899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:00.757899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:00.757899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:00.757899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:00.757899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:00.759802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:00.759802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:00.759802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:00.759802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:00.759802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:00.759802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:00.759802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:00.759802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:00.759802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:00.759802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:00.759802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:00.759802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:00.759802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:00.759802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:00.759802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:00.759802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:00.761346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:00.761346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:00.761346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:00.761346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:00.761346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:00.761346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:00.761346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:00.761346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:00.761346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:00.761346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:00.761346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 15:33:00.761346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:00.761346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 84/159] 10.222.0.102 () {56 vars in 1119 bytes} [Wed Aug 24 15:33:00 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:00.761346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db5194c-2f8d-4115-84f0-06997f73b3eb None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:00.835749 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f026f4-5aac-443b-8c38-e5c1eca0e27b None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:00.852455 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:00.892970 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd25915-c1c7-4c54-b8d5-d24a866da533 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:00.893888 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd25915-c1c7-4c54-b8d5-d24a866da533 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:00.894172 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd25915-c1c7-4c54-b8d5-d24a866da533 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:00.894449 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd25915-c1c7-4c54-b8d5-d24a866da533 None admin] PATH_INFO: `/v3/projects/749c6d651fc445cdb75db25a73873386` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:00.895752 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd25915-c1c7-4c54-b8d5-d24a866da533 None admin] RBAC: Authorizing `identity:get_project(project_id=749c6d651fc445cdb75db25a73873386)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:00.898123 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd25915-c1c7-4c54-b8d5-d24a866da533 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:00.899832 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 76/160] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:00 2022] GET /identity/v3/projects/749c6d651fc445cdb75db25a73873386 => generated 299 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:00.901067 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28db2e6-a4f4-4a88-a500-4dc177d212e4 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:00.901357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28db2e6-a4f4-4a88-a500-4dc177d212e4 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:00.901631 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28db2e6-a4f4-4a88-a500-4dc177d212e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:00.924419 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-c28db2e6-a4f4-4a88-a500-4dc177d212e4 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:00.925565 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-c28db2e6-a4f4-4a88-a500-4dc177d212e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:00.956544 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db5194c-2f8d-4115-84f0-06997f73b3eb None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:00.985688 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:01.004803 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273e588f-f636-45cf-b59b-8294dc6c1c11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:01.005135 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273e588f-f636-45cf-b59b-8294dc6c1c11 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:01.005418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273e588f-f636-45cf-b59b-8294dc6c1c11 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:01.005696 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273e588f-f636-45cf-b59b-8294dc6c1c11 None admin] PATH_INFO: `/v3/projects/749c6d651fc445cdb75db25a73873386` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:01.007055 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273e588f-f636-45cf-b59b-8294dc6c1c11 None admin] RBAC: Authorizing `identity:get_project(project_id=749c6d651fc445cdb75db25a73873386)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:01.008566 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273e588f-f636-45cf-b59b-8294dc6c1c11 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:01.010239 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 85/161] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:00 2022] GET /identity/v3/projects/749c6d651fc445cdb75db25a73873386 => generated 299 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:01.012817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273e588f-f636-45cf-b59b-8294dc6c1c11 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:01.023305 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 77/162] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:00 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:01.024503 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28db2e6-a4f4-4a88-a500-4dc177d212e4 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:01.159079 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28db2e6-a4f4-4a88-a500-4dc177d212e4 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:01.201685 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1b39e4-8f77-442e-9033-87543c8f18e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:01.203891 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1b39e4-8f77-442e-9033-87543c8f18e5 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:01.204343 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1b39e4-8f77-442e-9033-87543c8f18e5 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:01.204776 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1b39e4-8f77-442e-9033-87543c8f18e5 None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:01.205839 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1b39e4-8f77-442e-9033-87543c8f18e5 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:01.206290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1b39e4-8f77-442e-9033-87543c8f18e5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:01.208003 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1b39e4-8f77-442e-9033-87543c8f18e5 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:01.218027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 78/163] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:33:01 2022] GET /identity/v3/roles?name=anotherrole => generated 336 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:01.224791 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1b39e4-8f77-442e-9033-87543c8f18e5 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:01.263114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273e588f-f636-45cf-b59b-8294dc6c1c11 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:01.284891 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:01.329772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:01.343511 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1b39e4-8f77-442e-9033-87543c8f18e5 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:01.382783 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbb3b43-47ac-4031-bb8e-e7fd80634a75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:01.383654 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbb3b43-47ac-4031-bb8e-e7fd80634a75 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:01.384087 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbb3b43-47ac-4031-bb8e-e7fd80634a75 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:01.384594 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbb3b43-47ac-4031-bb8e-e7fd80634a75 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:01.385583 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbb3b43-47ac-4031-bb8e-e7fd80634a75 None admin] RBAC: Adding query filter params (role.id=b5d2ed3e20ab47eeb741b4f43ab2eebb, scope.project.id=749c6d651fc445cdb75db25a73873386, user.id=cb1c2ffc97364fa48e0d109484b7871c) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:01.386356 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbb3b43-47ac-4031-bb8e-e7fd80634a75 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:01.388081 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbb3b43-47ac-4031-bb8e-e7fd80634a75 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:01.396201 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 79/164] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:33:01 2022] GET /identity/v3/role_assignments?user.id=cb1c2ffc97364fa48e0d109484b7871c&scope.project.id=749c6d651fc445cdb75db25a73873386&role.id=b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 610 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:01.400686 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbb3b43-47ac-4031-bb8e-e7fd80634a75 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:01.401893 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f3d1ff-caff-4edf-b3cc-ff7875084380 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:01.402572 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f3d1ff-caff-4edf-b3cc-ff7875084380 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:01.402855 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f3d1ff-caff-4edf-b3cc-ff7875084380 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:01.403127 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f3d1ff-caff-4edf-b3cc-ff7875084380 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:01.444786 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f3d1ff-caff-4edf-b3cc-ff7875084380 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:01.449854 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-06f3d1ff-caff-4edf-b3cc-ff7875084380 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 15:33:01.449854 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:01.449854 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:01.449854 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:01.449854 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:01.449854 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:01.449854 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:01.449854 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:01.449854 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:01.449854 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:01.449854 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:01.449854 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:01.449854 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:01.449854 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:01.449854 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:01.449854 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:01.450868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:01.450868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:01.450868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:01.450868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:01.450868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:01.450868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:01.450868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:01.450868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:01.450868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:01.450868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:01.450868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:01.450868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:01.450868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:01.450868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:01.450868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:01.450868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:01.451863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:01.451863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:01.451863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:01.451863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:01.451863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:01.451863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:01.451863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:01.451863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:01.451863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:01.451863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:01.451863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 15:33:01.451863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:01.451863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 86/165] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:33:01 2022] GET /identity/v3/roles/reader => generated 84 bytes in 440 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:01.451863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f3d1ff-caff-4edf-b3cc-ff7875084380 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:01.544020 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbb3b43-47ac-4031-bb8e-e7fd80634a75 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:01.547980 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:01.586480 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d421c03-7e94-400d-8616-9075ce721031 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:01.587481 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d421c03-7e94-400d-8616-9075ce721031 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:01.588070 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d421c03-7e94-400d-8616-9075ce721031 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:01.588571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d421c03-7e94-400d-8616-9075ce721031 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:01.589519 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d421c03-7e94-400d-8616-9075ce721031 None admin] RBAC: Adding query filter params (role.id=4d8fbbaea81a4145b89af12350bb067c, scope.project.id=749c6d651fc445cdb75db25a73873386, user.id=3eb7904f17244b85baf19c438c53917f) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:01.590052 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d421c03-7e94-400d-8616-9075ce721031 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:01.591769 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d421c03-7e94-400d-8616-9075ce721031 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:01.600894 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 80/166] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:33:01 2022] GET /identity/v3/role_assignments?user.id=3eb7904f17244b85baf19c438c53917f&scope.project.id=749c6d651fc445cdb75db25a73873386&role.id=4d8fbbaea81a4145b89af12350bb067c => generated 610 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:01.602203 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d421c03-7e94-400d-8616-9075ce721031 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:01.639479 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f3d1ff-caff-4edf-b3cc-ff7875084380 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:01.708855 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5688795e-094e-4f52-8438-ad72b1620a92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:01.710662 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5688795e-094e-4f52-8438-ad72b1620a92 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:01.712639 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5688795e-094e-4f52-8438-ad72b1620a92 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:01.713166 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5688795e-094e-4f52-8438-ad72b1620a92 None admin] PATH_INFO: `/v3/users/cb1c2ffc97364fa48e0d109484b7871c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:01.716360 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5688795e-094e-4f52-8438-ad72b1620a92 None admin] RBAC: Authorizing `identity:get_user(user_id=cb1c2ffc97364fa48e0d109484b7871c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:01.720077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5688795e-094e-4f52-8438-ad72b1620a92 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:01.725510 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 87/167] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:33:01 2022] GET /identity/v3/users/cb1c2ffc97364fa48e0d109484b7871c => generated 276 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:01.728253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5688795e-094e-4f52-8438-ad72b1620a92 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:01.759907 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d421c03-7e94-400d-8616-9075ce721031 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:01.837173 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:01.837477 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ed4b99-d03a-49bc-bc9e-d178cecc0d8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:01.838372 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ed4b99-d03a-49bc-bc9e-d178cecc0d8d None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:01.838372 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ed4b99-d03a-49bc-bc9e-d178cecc0d8d None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:01.838704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ed4b99-d03a-49bc-bc9e-d178cecc0d8d None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:01.839430 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ed4b99-d03a-49bc-bc9e-d178cecc0d8d None admin] RBAC: Adding query filter params (name=reader) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:01.840013 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ed4b99-d03a-49bc-bc9e-d178cecc0d8d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:01.843048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ed4b99-d03a-49bc-bc9e-d178cecc0d8d None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:01.864364 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 81/168] 10.222.0.102 () {56 vars in 1106 bytes} [Wed Aug 24 15:33:01 2022] GET /identity/v3/roles?name=reader => generated 343 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:01.865249 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ed4b99-d03a-49bc-bc9e-d178cecc0d8d None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:01.907849 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:01.949766 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5688795e-094e-4f52-8438-ad72b1620a92 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:02.006082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ed4b99-d03a-49bc-bc9e-d178cecc0d8d None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:02.018641 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73adc4ad-b969-4651-97d3-d55d4211eddb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:02.019361 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73adc4ad-b969-4651-97d3-d55d4211eddb None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:02.019658 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73adc4ad-b969-4651-97d3-d55d4211eddb None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:02.019942 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73adc4ad-b969-4651-97d3-d55d4211eddb None admin] PATH_INFO: `/v3/roles/member` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:02.044240 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73adc4ad-b969-4651-97d3-d55d4211eddb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:02.051522 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cff4d12-951f-4d73-9a5a-a379d9c62b55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:02.052360 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-73adc4ad-b969-4651-97d3-d55d4211eddb None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:33:02.052360 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:02.052360 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:02.052360 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:02.052360 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:02.052360 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:02.052360 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:02.052360 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:02.052360 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:02.052360 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:02.052360 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:02.052360 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:02.052360 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:02.052360 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:02.052360 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:02.052360 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:02.053226 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:02.053226 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:02.053226 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:02.053226 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:02.053226 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:02.053226 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:02.053226 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:02.053226 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:02.053226 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:02.053226 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:02.053226 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:02.053226 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:02.053226 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:02.053226 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:02.053226 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:02.053226 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:02.054002 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:02.054002 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:02.054002 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:02.054002 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:02.054002 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:02.054002 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:02.054002 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:02.054002 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:02.054002 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:02.054002 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:02.054002 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:33:02.054002 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:02.054002 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 88/169] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:33:01 2022] GET /identity/v3/roles/member => generated 84 bytes in 326 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:02.054647 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73adc4ad-b969-4651-97d3-d55d4211eddb None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:02.055749 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cff4d12-951f-4d73-9a5a-a379d9c62b55 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:02.056336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cff4d12-951f-4d73-9a5a-a379d9c62b55 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:02.056809 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cff4d12-951f-4d73-9a5a-a379d9c62b55 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:02.135319 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cff4d12-951f-4d73-9a5a-a379d9c62b55 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:02.137381 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cff4d12-951f-4d73-9a5a-a379d9c62b55 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:02.167556 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:02.235218 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73adc4ad-b969-4651-97d3-d55d4211eddb None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:02.251057 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:02.258197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 82/170] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:33:01 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:33:02.259384 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cff4d12-951f-4d73-9a5a-a379d9c62b55 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:02.286652 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d56296-c5b8-492a-bbe0-5b99bddbebf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:02.287403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d56296-c5b8-492a-bbe0-5b99bddbebf1 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:02.287587 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d56296-c5b8-492a-bbe0-5b99bddbebf1 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:02.287895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d56296-c5b8-492a-bbe0-5b99bddbebf1 None admin] PATH_INFO: `/v3/users/ed4c3aa4c7b14a35b57177bf48a0e250` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:02.289080 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d56296-c5b8-492a-bbe0-5b99bddbebf1 None admin] RBAC: Authorizing `identity:get_user(user_id=ed4c3aa4c7b14a35b57177bf48a0e250)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:02.290548 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d56296-c5b8-492a-bbe0-5b99bddbebf1 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:02.292548 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 89/171] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:33:02 2022] GET /identity/v3/users/ed4c3aa4c7b14a35b57177bf48a0e250 => generated 290 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:02.293340 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d56296-c5b8-492a-bbe0-5b99bddbebf1 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:02.384630 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cff4d12-951f-4d73-9a5a-a379d9c62b55 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:02.404890 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d56296-c5b8-492a-bbe0-5b99bddbebf1 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:02.416945 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:02.426206 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4362e528-2d46-4194-b37a-f14150ed5034 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:02.427314 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4362e528-2d46-4194-b37a-f14150ed5034 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:02.427921 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4362e528-2d46-4194-b37a-f14150ed5034 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:02.428382 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4362e528-2d46-4194-b37a-f14150ed5034 None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:02.429268 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4362e528-2d46-4194-b37a-f14150ed5034 None admin] RBAC: Adding query filter params (name=member) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:02.429781 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4362e528-2d46-4194-b37a-f14150ed5034 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:02.431601 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4362e528-2d46-4194-b37a-f14150ed5034 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:02.443220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 83/172] 10.222.0.102 () {56 vars in 1106 bytes} [Wed Aug 24 15:33:02 2022] GET /identity/v3/roles?name=member => generated 343 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:02.444536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4362e528-2d46-4194-b37a-f14150ed5034 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:02.444859 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ccd1525-8974-4173-b6da-d629a8ac283b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:02.445576 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ccd1525-8974-4173-b6da-d629a8ac283b None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:02.445926 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ccd1525-8974-4173-b6da-d629a8ac283b None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:02.446267 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ccd1525-8974-4173-b6da-d629a8ac283b None admin] PATH_INFO: `/v3/projects/749c6d651fc445cdb75db25a73873386` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:02.447587 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ccd1525-8974-4173-b6da-d629a8ac283b None admin] RBAC: Authorizing `identity:get_project(project_id=749c6d651fc445cdb75db25a73873386)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:02.449254 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ccd1525-8974-4173-b6da-d629a8ac283b None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:02.451043 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 90/173] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:02 2022] GET /identity/v3/projects/749c6d651fc445cdb75db25a73873386 => generated 299 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:02.452936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ccd1525-8974-4173-b6da-d629a8ac283b None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:02.504224 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:02.564192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ccd1525-8974-4173-b6da-d629a8ac283b None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:02.611854 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46bc239b-9758-4fa4-94ac-7f78e89fea51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:02.612599 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bc239b-9758-4fa4-94ac-7f78e89fea51 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:02.613152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bc239b-9758-4fa4-94ac-7f78e89fea51 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:02.613260 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bc239b-9758-4fa4-94ac-7f78e89fea51 None admin] PATH_INFO: `/v3/users/cb1c2ffc97364fa48e0d109484b7871c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:02.614888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46bc239b-9758-4fa4-94ac-7f78e89fea51 None admin] RBAC: Authorizing `identity:get_user(user_id=cb1c2ffc97364fa48e0d109484b7871c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:02.616720 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46bc239b-9758-4fa4-94ac-7f78e89fea51 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:02.618362 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 91/174] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:33:02 2022] GET /identity/v3/users/cb1c2ffc97364fa48e0d109484b7871c => generated 276 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:02.619280 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46bc239b-9758-4fa4-94ac-7f78e89fea51 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:02.656103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4362e528-2d46-4194-b37a-f14150ed5034 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:02.729197 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46bc239b-9758-4fa4-94ac-7f78e89fea51 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:02.745618 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:02.767325 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292abfee-9d7b-4a1c-bb5e-3ec164e70448 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:02.767997 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292abfee-9d7b-4a1c-bb5e-3ec164e70448 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:02.768281 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292abfee-9d7b-4a1c-bb5e-3ec164e70448 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:02.768662 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292abfee-9d7b-4a1c-bb5e-3ec164e70448 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:02.769432 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292abfee-9d7b-4a1c-bb5e-3ec164e70448 None admin] RBAC: Adding query filter params (role.id=b8429ebd9c2a4cbbbbf30ad56ce23bfa, scope.project.id=749c6d651fc445cdb75db25a73873386, user.id=cb1c2ffc97364fa48e0d109484b7871c) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:02.769749 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292abfee-9d7b-4a1c-bb5e-3ec164e70448 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:02.771247 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292abfee-9d7b-4a1c-bb5e-3ec164e70448 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:02.772391 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff30990c-c525-4aba-a5dc-3626c91ea7e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:02.773049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff30990c-c525-4aba-a5dc-3626c91ea7e8 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:02.773334 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff30990c-c525-4aba-a5dc-3626c91ea7e8 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:02.773612 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff30990c-c525-4aba-a5dc-3626c91ea7e8 None admin] PATH_INFO: `/v3/projects/749c6d651fc445cdb75db25a73873386` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:02.776000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff30990c-c525-4aba-a5dc-3626c91ea7e8 None admin] RBAC: Authorizing `identity:get_project(project_id=749c6d651fc445cdb75db25a73873386)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:02.777571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff30990c-c525-4aba-a5dc-3626c91ea7e8 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:02.778932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 92/175] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:33:02 2022] GET /identity/v3/role_assignments?user.id=cb1c2ffc97364fa48e0d109484b7871c&scope.project.id=749c6d651fc445cdb75db25a73873386&role.id=b8429ebd9c2a4cbbbbf30ad56ce23bfa => generated 610 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:02.779478 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 84/176] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:02 2022] GET /identity/v3/projects/749c6d651fc445cdb75db25a73873386 => generated 299 bytes in 335 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:02.780424 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292abfee-9d7b-4a1c-bb5e-3ec164e70448 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:02.786675 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff30990c-c525-4aba-a5dc-3626c91ea7e8 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:02.905431 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292abfee-9d7b-4a1c-bb5e-3ec164e70448 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:02.947822 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e1d9a18-f20f-4256-8b3f-499a7f7a3d4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:02.948892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1d9a18-f20f-4256-8b3f-499a7f7a3d4b None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:02.949499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1d9a18-f20f-4256-8b3f-499a7f7a3d4b None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:02.950005 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1d9a18-f20f-4256-8b3f-499a7f7a3d4b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:02.992787 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:03.006250 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:03.031765 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e1d9a18-f20f-4256-8b3f-499a7f7a3d4b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:03.034460 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e1d9a18-f20f-4256-8b3f-499a7f7a3d4b None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:03.097549 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff30990c-c525-4aba-a5dc-3626c91ea7e8 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:03.150369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 93/177] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:33:02 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:33:03.152384 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960644c8-47e4-4471-985b-96a5e7a8ea5b None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:03.152863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960644c8-47e4-4471-985b-96a5e7a8ea5b None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:03.152984 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9659cac7-208e-4696-b9a8-012d3b1f24d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:03.153459 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960644c8-47e4-4471-985b-96a5e7a8ea5b None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:03.153718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9659cac7-208e-4696-b9a8-012d3b1f24d3 None admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:03.154003 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9659cac7-208e-4696-b9a8-012d3b1f24d3 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:03.154278 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9659cac7-208e-4696-b9a8-012d3b1f24d3 None admin] PATH_INFO: `/v3/projects/749c6d651fc445cdb75db25a73873386/users/ed4c3aa4c7b14a35b57177bf48a0e250/roles/868817cab1414b5aa207bc5bdf4f1d9a` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:03.155751 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 94/178] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:03 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:03.157298 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-960644c8-47e4-4471-985b-96a5e7a8ea5b None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:03.157554 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9659cac7-208e-4696-b9a8-012d3b1f24d3 None admin] RBAC: Authorizing `identity:create_grant(project_id=749c6d651fc445cdb75db25a73873386, user_id=ed4c3aa4c7b14a35b57177bf48a0e250, role_id=868817cab1414b5aa207bc5bdf4f1d9a)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:03.159508 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9659cac7-208e-4696-b9a8-012d3b1f24d3 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:03.176127 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 85/179] 10.222.0.102 () {58 vars in 1444 bytes} [Wed Aug 24 15:33:02 2022] PUT /identity/v3/projects/749c6d651fc445cdb75db25a73873386/users/ed4c3aa4c7b14a35b57177bf48a0e250/roles/868817cab1414b5aa207bc5bdf4f1d9a => generated 0 bytes in 390 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:33:03.177482 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149e0607-516e-40bf-9a49-54d89f2a06bc None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:03.177798 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149e0607-516e-40bf-9a49-54d89f2a06bc None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:03.178117 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149e0607-516e-40bf-9a49-54d89f2a06bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:03.226719 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-149e0607-516e-40bf-9a49-54d89f2a06bc None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:03.228034 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-149e0607-516e-40bf-9a49-54d89f2a06bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:03.261535 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:03.289751 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-960644c8-47e4-4471-985b-96a5e7a8ea5b None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:03.339891 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b526f927-7bdb-4297-9157-df587a805f6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:03.341987 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b526f927-7bdb-4297-9157-df587a805f6c None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:03.343220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b526f927-7bdb-4297-9157-df587a805f6c None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:03.344344 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b526f927-7bdb-4297-9157-df587a805f6c None admin] PATH_INFO: `/v3/projects/2dcca3c9f7d64026b4e21bfca99e2ab8` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:03.347389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b526f927-7bdb-4297-9157-df587a805f6c None admin] RBAC: Authorizing `identity:get_project(project_id=2dcca3c9f7d64026b4e21bfca99e2ab8)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:03.349290 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b526f927-7bdb-4297-9157-df587a805f6c None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:03.352173 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 95/180] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:03 2022] GET /identity/v3/projects/2dcca3c9f7d64026b4e21bfca99e2ab8 => generated 313 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:03.360389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b526f927-7bdb-4297-9157-df587a805f6c None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:03.364943 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 86/181] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:03 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:03.492727 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b526f927-7bdb-4297-9157-df587a805f6c None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:03.529813 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2c3ca5-8747-4a78-829e-9f9854c46e2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:03.530753 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2c3ca5-8747-4a78-829e-9f9854c46e2f None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:03.531776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2c3ca5-8747-4a78-829e-9f9854c46e2f None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:03.532389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2c3ca5-8747-4a78-829e-9f9854c46e2f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:03.533297 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2c3ca5-8747-4a78-829e-9f9854c46e2f None admin] RBAC: Adding query filter params (role.id=b5d2ed3e20ab47eeb741b4f43ab2eebb, scope.project.id=2dcca3c9f7d64026b4e21bfca99e2ab8, user.id=cb1c2ffc97364fa48e0d109484b7871c) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:03.533778 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2c3ca5-8747-4a78-829e-9f9854c46e2f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:03.535505 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2c3ca5-8747-4a78-829e-9f9854c46e2f None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:03.542287 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 96/182] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:33:03 2022] GET /identity/v3/role_assignments?user.id=cb1c2ffc97364fa48e0d109484b7871c&scope.project.id=2dcca3c9f7d64026b4e21bfca99e2ab8&role.id=b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 610 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:33:03.700060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149e0607-516e-40bf-9a49-54d89f2a06bc None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:03.762578 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:03.809154 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3448a876-48ea-462a-88df-1d59ae44f1a4 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:03.809659 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3448a876-48ea-462a-88df-1d59ae44f1a4 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:03.809659 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3448a876-48ea-462a-88df-1d59ae44f1a4 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:03.811274 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 97/183] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:03 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:33:03.818867 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e0ef73-4f61-4df3-946d-7bf56f3c2976 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:03.819480 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e0ef73-4f61-4df3-946d-7bf56f3c2976 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:03.819985 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e0ef73-4f61-4df3-946d-7bf56f3c2976 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:03.852853 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149e0607-516e-40bf-9a49-54d89f2a06bc None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:03.855717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-75e0ef73-4f61-4df3-946d-7bf56f3c2976 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:03.857156 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-75e0ef73-4f61-4df3-946d-7bf56f3c2976 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:03.860360 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:03.934391 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c7237d-cb09-4ca2-a78b-5376c70f6b11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:03.935075 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c7237d-cb09-4ca2-a78b-5376c70f6b11 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:03.935354 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c7237d-cb09-4ca2-a78b-5376c70f6b11 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:03.935660 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c7237d-cb09-4ca2-a78b-5376c70f6b11 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:03.956546 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c7237d-cb09-4ca2-a78b-5376c70f6b11 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:03.961336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-09c7237d-cb09-4ca2-a78b-5376c70f6b11 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:33:03.961336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:03.961336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:03.961336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:03.961336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:03.961336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:03.961336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:03.961336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:03.961336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:03.961336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:03.961336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:03.961336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:03.961336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:03.961336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:03.961336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:03.961336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:03.962277 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:03.962277 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:03.962277 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:03.962277 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:03.962277 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:03.962277 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:03.962277 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:03.962277 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:03.962277 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:03.962277 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:03.962277 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:03.962277 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:03.962277 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:03.962277 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:03.962277 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:03.962277 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:03.963263 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:03.963263 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:03.963263 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:03.963263 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:03.963263 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:03.963263 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:03.963263 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:03.963263 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:03.963263 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:03.963263 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:03.963263 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:33:03.963263 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:03.963263 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 87/184] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:33:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 263 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:03.965598 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015890a8-878e-48d4-985f-7daf92d231a6 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:03.965953 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015890a8-878e-48d4-985f-7daf92d231a6 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:03.966303 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015890a8-878e-48d4-985f-7daf92d231a6 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:03.967879 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 88/185] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:03 2022] GET /identity => generated 271 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:03.972581 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 98/186] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:03 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:03.973384 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015890a8-878e-48d4-985f-7daf92d231a6 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:03.974169 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697abf79-e6b9-4aad-99aa-a5a0c9d51023 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:03.974412 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697abf79-e6b9-4aad-99aa-a5a0c9d51023 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:03.974935 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697abf79-e6b9-4aad-99aa-a5a0c9d51023 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:03.993829 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-697abf79-e6b9-4aad-99aa-a5a0c9d51023 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:03.995123 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-697abf79-e6b9-4aad-99aa-a5a0c9d51023 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:04.074540 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 99/187] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:03 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:04.116702 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015890a8-878e-48d4-985f-7daf92d231a6 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:04.124557 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:04.154189 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb04492-f557-4900-aef9-1a9a0af38ccf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:04.154793 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb04492-f557-4900-aef9-1a9a0af38ccf None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:04.155068 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb04492-f557-4900-aef9-1a9a0af38ccf None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:04.155339 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb04492-f557-4900-aef9-1a9a0af38ccf None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:04.156076 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb04492-f557-4900-aef9-1a9a0af38ccf None admin] RBAC: Adding query filter params (name=admin) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:04.156378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb04492-f557-4900-aef9-1a9a0af38ccf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:04.157765 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb04492-f557-4900-aef9-1a9a0af38ccf None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:04.166796 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 89/188] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:33:03 2022] GET /identity/v3/roles?name=admin => generated 341 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:04.172806 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697abf79-e6b9-4aad-99aa-a5a0c9d51023 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:04.245570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:04.276775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb04492-f557-4900-aef9-1a9a0af38ccf None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:04.292609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697abf79-e6b9-4aad-99aa-a5a0c9d51023 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:04.335506 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f208356f-b3b2-4a43-8198-bd2e89c20a13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:04.336471 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f208356f-b3b2-4a43-8198-bd2e89c20a13 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:04.336965 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f208356f-b3b2-4a43-8198-bd2e89c20a13 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:04.337511 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f208356f-b3b2-4a43-8198-bd2e89c20a13 None admin] PATH_INFO: `/v3/users/bb05bf01d9d24a5296517f8bde87462b` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:04.353751 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f208356f-b3b2-4a43-8198-bd2e89c20a13 None admin] RBAC: Authorizing `identity:get_user(user_id=bb05bf01d9d24a5296517f8bde87462b)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:04.355235 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f208356f-b3b2-4a43-8198-bd2e89c20a13 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:04.357100 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 100/189] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:33:04 2022] GET /identity/v3/users/bb05bf01d9d24a5296517f8bde87462b => generated 284 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:04.358581 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f208356f-b3b2-4a43-8198-bd2e89c20a13 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:04.387238 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb04492-f557-4900-aef9-1a9a0af38ccf None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:04.426285 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d66b978-0d52-462a-8f26-a3f6a3403065 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:04.426915 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d66b978-0d52-462a-8f26-a3f6a3403065 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:04.427198 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d66b978-0d52-462a-8f26-a3f6a3403065 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:04.427472 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d66b978-0d52-462a-8f26-a3f6a3403065 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:04.439456 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d66b978-0d52-462a-8f26-a3f6a3403065 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:04.445171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-3d66b978-0d52-462a-8f26-a3f6a3403065 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 15:33:04.445171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:04.445171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:04.445171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:04.445171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:04.445171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:04.445171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:04.445171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:04.445171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:04.445171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:04.445171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:04.445171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:04.445171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:04.445171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:04.445171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:04.445171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:04.446063 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:04.446063 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:04.446063 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:04.446063 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:04.446063 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:04.446063 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:04.446063 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:04.446063 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:04.446063 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:04.446063 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:04.446063 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:04.446063 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:04.446063 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:04.446063 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:04.446063 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:04.446063 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:04.446860 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:04.446860 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:04.446860 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:04.446860 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:04.446860 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:04.446860 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:04.446860 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:04.446860 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:04.446860 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:04.446860 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:04.446860 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 15:33:04.446860 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:04.446860 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 90/190] 10.222.0.102 () {56 vars in 1119 bytes} [Wed Aug 24 15:33:04 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:33:04.451758 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d66b978-0d52-462a-8f26-a3f6a3403065 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:04.469792 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f208356f-b3b2-4a43-8198-bd2e89c20a13 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:04.482675 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:04.504181 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:04.514837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1dafb3-f1c7-4a87-962e-de55b9e09e77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:04.516375 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1dafb3-f1c7-4a87-962e-de55b9e09e77 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:04.516700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1dafb3-f1c7-4a87-962e-de55b9e09e77 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:04.516926 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1dafb3-f1c7-4a87-962e-de55b9e09e77 None admin] PATH_INFO: `/v3/domains/default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:04.518065 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b1dafb3-f1c7-4a87-962e-de55b9e09e77 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:04.519506 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b1dafb3-f1c7-4a87-962e-de55b9e09e77 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:04.525591 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 101/191] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:33:04 2022] GET /identity/v3/domains/default => generated 200 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:04.525591 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1dafb3-f1c7-4a87-962e-de55b9e09e77 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:04.570178 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d66b978-0d52-462a-8f26-a3f6a3403065 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:04.610811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b286a9a9-580e-4106-b47d-b9a06b725e81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:04.611426 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b286a9a9-580e-4106-b47d-b9a06b725e81 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:04.611732 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b286a9a9-580e-4106-b47d-b9a06b725e81 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:04.611996 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b286a9a9-580e-4106-b47d-b9a06b725e81 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:04.632909 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1dafb3-f1c7-4a87-962e-de55b9e09e77 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:04.672213 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03c54f7-8761-42c0-8bef-09df5ccc382f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:04.672847 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03c54f7-8761-42c0-8bef-09df5ccc382f None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:04.673125 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03c54f7-8761-42c0-8bef-09df5ccc382f None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:04.673397 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03c54f7-8761-42c0-8bef-09df5ccc382f None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:04.674193 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e03c54f7-8761-42c0-8bef-09df5ccc382f None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:04.674740 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e03c54f7-8761-42c0-8bef-09df5ccc382f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:04.677372 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e03c54f7-8761-42c0-8bef-09df5ccc382f None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:04.690530 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 102/192] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:33:04 2022] GET /identity/v3/roles?name=anotherrole => generated 336 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:04.691789 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03c54f7-8761-42c0-8bef-09df5ccc382f None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:04.700051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b286a9a9-580e-4106-b47d-b9a06b725e81 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:04.701815 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b286a9a9-580e-4106-b47d-b9a06b725e81 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:04.736969 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:04.772470 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:04.820904 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03c54f7-8761-42c0-8bef-09df5ccc382f None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:04.830185 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 91/193] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:33:04 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:33:04.830185 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b286a9a9-580e-4106-b47d-b9a06b725e81 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:04.867962 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b474bf-f3a0-4263-893b-076a95611b7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:04.868577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b474bf-f3a0-4263-893b-076a95611b7f None admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:04.868861 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b474bf-f3a0-4263-893b-076a95611b7f None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:04.869840 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b474bf-f3a0-4263-893b-076a95611b7f None admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:04.869840 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b474bf-f3a0-4263-893b-076a95611b7f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:04.871229 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b474bf-f3a0-4263-893b-076a95611b7f None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:04.901597 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 103/194] 10.222.0.102 () {60 vars in 1137 bytes} [Wed Aug 24 15:33:04 2022] POST /identity/v3/users => generated 298 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:33:04.902696 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b474bf-f3a0-4263-893b-076a95611b7f None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:04.976897 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b286a9a9-580e-4106-b47d-b9a06b725e81 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:05.017192 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:05.028811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b866dfd3-7f7b-4cc6-a8f4-18ef78be9b8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:05.029357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b866dfd3-7f7b-4cc6-a8f4-18ef78be9b8f None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:05.029640 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b866dfd3-7f7b-4cc6-a8f4-18ef78be9b8f None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:05.029918 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b866dfd3-7f7b-4cc6-a8f4-18ef78be9b8f None admin] PATH_INFO: `/v3/users/bb05bf01d9d24a5296517f8bde87462b` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:05.032018 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b866dfd3-7f7b-4cc6-a8f4-18ef78be9b8f None admin] RBAC: Authorizing `identity:get_user(user_id=bb05bf01d9d24a5296517f8bde87462b)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:05.034113 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b866dfd3-7f7b-4cc6-a8f4-18ef78be9b8f None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:05.039835 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 92/195] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:33:04 2022] GET /identity/v3/users/bb05bf01d9d24a5296517f8bde87462b => generated 284 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:05.042282 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b866dfd3-7f7b-4cc6-a8f4-18ef78be9b8f None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:05.065106 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b474bf-f3a0-4263-893b-076a95611b7f None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:05.084652 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:05.123406 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41fd1f3-36c5-4b6f-bd37-dbef85d550f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:05.124218 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41fd1f3-36c5-4b6f-bd37-dbef85d550f7 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:05.124504 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41fd1f3-36c5-4b6f-bd37-dbef85d550f7 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:05.125420 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41fd1f3-36c5-4b6f-bd37-dbef85d550f7 None admin] PATH_INFO: `/v3/projects/86a6b464bc2d4f72895f92c759e26a36` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:05.127675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b41fd1f3-36c5-4b6f-bd37-dbef85d550f7 None admin] RBAC: Authorizing `identity:get_project(project_id=86a6b464bc2d4f72895f92c759e26a36)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:05.128900 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b41fd1f3-36c5-4b6f-bd37-dbef85d550f7 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:05.131008 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 104/196] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:04 2022] GET /identity/v3/projects/86a6b464bc2d4f72895f92c759e26a36 => generated 303 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:05.137632 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41fd1f3-36c5-4b6f-bd37-dbef85d550f7 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:05.207121 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b866dfd3-7f7b-4cc6-a8f4-18ef78be9b8f None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:05.248991 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a25d17a1-c7ef-4d87-9883-cad03c4dcada None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:05.249640 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25d17a1-c7ef-4d87-9883-cad03c4dcada None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:05.250109 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25d17a1-c7ef-4d87-9883-cad03c4dcada None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:05.250468 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25d17a1-c7ef-4d87-9883-cad03c4dcada None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:05.279449 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41fd1f3-36c5-4b6f-bd37-dbef85d550f7 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:05.327143 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ef3144-5551-42b9-b027-6ecf5d2b4cfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:05.328916 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ef3144-5551-42b9-b027-6ecf5d2b4cfa None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:05.329449 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ef3144-5551-42b9-b027-6ecf5d2b4cfa None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:05.329918 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ef3144-5551-42b9-b027-6ecf5d2b4cfa None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:05.331387 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84ef3144-5551-42b9-b027-6ecf5d2b4cfa None admin] RBAC: Adding query filter params (role.id=4d8fbbaea81a4145b89af12350bb067c, scope.project.id=86a6b464bc2d4f72895f92c759e26a36, user.id=bb05bf01d9d24a5296517f8bde87462b) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:05.332199 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84ef3144-5551-42b9-b027-6ecf5d2b4cfa None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:05.334367 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84ef3144-5551-42b9-b027-6ecf5d2b4cfa None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:05.337502 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:05.342743 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 105/197] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:33:05 2022] GET /identity/v3/role_assignments?user.id=bb05bf01d9d24a5296517f8bde87462b&scope.project.id=86a6b464bc2d4f72895f92c759e26a36&role.id=4d8fbbaea81a4145b89af12350bb067c => generated 260 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:33:05.359794 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a25d17a1-c7ef-4d87-9883-cad03c4dcada None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:05.360668 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a25d17a1-c7ef-4d87-9883-cad03c4dcada None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:05.490661 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 93/198] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:33:05 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 449 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:33:05.496778 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ef3144-5551-42b9-b027-6ecf5d2b4cfa None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:05.552067 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:05.609138 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ef3144-5551-42b9-b027-6ecf5d2b4cfa None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:05.625968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9999ecc2-e296-4a19-a455-6b9bd441139f None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:05.626270 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9999ecc2-e296-4a19-a455-6b9bd441139f None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:05.626527 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9999ecc2-e296-4a19-a455-6b9bd441139f None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:05.628054 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 94/199] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:05 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:33:05.633100 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34eedd0f-c4d6-4d05-8a5a-1d33c0f125d8 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:05.633371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34eedd0f-c4d6-4d05-8a5a-1d33c0f125d8 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:05.633649 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34eedd0f-c4d6-4d05-8a5a-1d33c0f125d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:05.646419 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a45b8a-48bb-4567-be95-c4cb12ea6ee2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:05.647062 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a45b8a-48bb-4567-be95-c4cb12ea6ee2 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:05.647344 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a45b8a-48bb-4567-be95-c4cb12ea6ee2 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:05.647617 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a45b8a-48bb-4567-be95-c4cb12ea6ee2 None admin] PATH_INFO: `/v3/projects/86a6b464bc2d4f72895f92c759e26a36` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:05.648790 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a45b8a-48bb-4567-be95-c4cb12ea6ee2 None admin] RBAC: Authorizing `identity:get_project(project_id=86a6b464bc2d4f72895f92c759e26a36)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:05.650248 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a45b8a-48bb-4567-be95-c4cb12ea6ee2 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:05.655716 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 106/200] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:05 2022] GET /identity/v3/projects/86a6b464bc2d4f72895f92c759e26a36 => generated 303 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:05.657708 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-34eedd0f-c4d6-4d05-8a5a-1d33c0f125d8 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:05.658921 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a45b8a-48bb-4567-be95-c4cb12ea6ee2 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:05.659339 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-34eedd0f-c4d6-4d05-8a5a-1d33c0f125d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:05.769861 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a45b8a-48bb-4567-be95-c4cb12ea6ee2 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:05.785168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:05.809452 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-989f5d97-924a-48dd-a722-b5de6d2720d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:05.810091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989f5d97-924a-48dd-a722-b5de6d2720d9 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:05.810444 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989f5d97-924a-48dd-a722-b5de6d2720d9 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:05.810842 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989f5d97-924a-48dd-a722-b5de6d2720d9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:05.811825 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-989f5d97-924a-48dd-a722-b5de6d2720d9 None admin] RBAC: Adding query filter params (role.id=b8429ebd9c2a4cbbbbf30ad56ce23bfa, scope.project.id=86a6b464bc2d4f72895f92c759e26a36, user.id=bb05bf01d9d24a5296517f8bde87462b) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:05.812635 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-989f5d97-924a-48dd-a722-b5de6d2720d9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:05.814822 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-989f5d97-924a-48dd-a722-b5de6d2720d9 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:05.823759 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 107/201] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:33:05 2022] GET /identity/v3/role_assignments?user.id=bb05bf01d9d24a5296517f8bde87462b&scope.project.id=86a6b464bc2d4f72895f92c759e26a36&role.id=b8429ebd9c2a4cbbbbf30ad56ce23bfa => generated 260 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:33:05.869839 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 95/202] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:05 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 237 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 15:33:06.252331 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-989f5d97-924a-48dd-a722-b5de6d2720d9 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:06.386244 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-989f5d97-924a-48dd-a722-b5de6d2720d9 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:06.424630 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0cfe3a-cb15-4257-9369-9926e910af04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:06.425289 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0cfe3a-cb15-4257-9369-9926e910af04 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:06.425573 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0cfe3a-cb15-4257-9369-9926e910af04 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:06.425850 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0cfe3a-cb15-4257-9369-9926e910af04 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:06.436741 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d0cfe3a-cb15-4257-9369-9926e910af04 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:06.443220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-0d0cfe3a-cb15-4257-9369-9926e910af04 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 15:33:06.443220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:06.443220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:06.443220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:06.443220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:06.443220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:06.443220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:06.443220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:06.443220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:06.443220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:06.443220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:06.443220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:06.443220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:06.443220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:06.443220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:06.443220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:06.444114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:06.444114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:06.444114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:06.444114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:06.444114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:06.444114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:06.444114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:06.444114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:06.444114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:06.444114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:06.444114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:06.444114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:06.444114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:06.444114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:06.444114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:06.444114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:06.444899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:06.444899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:06.444899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:06.444899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:06.444899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:06.444899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:06.444899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:06.444899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:06.444899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:06.444899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:06.444899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 15:33:06.444899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:06.444899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 108/203] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:33:06 2022] GET /identity/v3/roles/reader => generated 84 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:06.450217 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34eedd0f-c4d6-4d05-8a5a-1d33c0f125d8 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:06.506389 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:06.602465 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34eedd0f-c4d6-4d05-8a5a-1d33c0f125d8 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:06.650348 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ea6c4d-0f37-4685-bc83-23f567d2e0f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:06.650969 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ea6c4d-0f37-4685-bc83-23f567d2e0f9 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:06.651254 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ea6c4d-0f37-4685-bc83-23f567d2e0f9 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:06.651529 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ea6c4d-0f37-4685-bc83-23f567d2e0f9 None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:06.652233 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ea6c4d-0f37-4685-bc83-23f567d2e0f9 None admin] RBAC: Adding query filter params (name=reader) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:06.652541 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ea6c4d-0f37-4685-bc83-23f567d2e0f9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:06.653950 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ea6c4d-0f37-4685-bc83-23f567d2e0f9 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:06.663926 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 96/204] 10.222.0.102 () {56 vars in 1106 bytes} [Wed Aug 24 15:33:06 2022] GET /identity/v3/roles?name=reader => generated 343 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:06.673793 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0cfe3a-cb15-4257-9369-9926e910af04 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:06.741011 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:06.819529 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0cfe3a-cb15-4257-9369-9926e910af04 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:06.854849 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f864b13b-db9e-42a2-81cd-bfa989374494 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:06.855486 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f864b13b-db9e-42a2-81cd-bfa989374494 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:06.855797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f864b13b-db9e-42a2-81cd-bfa989374494 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:06.856122 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f864b13b-db9e-42a2-81cd-bfa989374494 None admin] PATH_INFO: `/v3/users/ed4c3aa4c7b14a35b57177bf48a0e250` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:06.857354 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f864b13b-db9e-42a2-81cd-bfa989374494 None admin] RBAC: Authorizing `identity:get_user(user_id=ed4c3aa4c7b14a35b57177bf48a0e250)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:06.858926 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f864b13b-db9e-42a2-81cd-bfa989374494 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:06.860856 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 109/205] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:33:06 2022] GET /identity/v3/users/ed4c3aa4c7b14a35b57177bf48a0e250 => generated 290 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:06.866438 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ea6c4d-0f37-4685-bc83-23f567d2e0f9 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:07.013379 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:07.034006 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ea6c4d-0f37-4685-bc83-23f567d2e0f9 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:07.087562 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3c5997-5c68-42a3-98c8-808bd73614e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:07.088120 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3c5997-5c68-42a3-98c8-808bd73614e3 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:07.088407 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3c5997-5c68-42a3-98c8-808bd73614e3 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:07.088680 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3c5997-5c68-42a3-98c8-808bd73614e3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:07.188343 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c3c5997-5c68-42a3-98c8-808bd73614e3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:07.190276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c3c5997-5c68-42a3-98c8-808bd73614e3 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:07.312751 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:07.333146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 97/206] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:33:06 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 467 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:33:07.340811 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f864b13b-db9e-42a2-81cd-bfa989374494 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:07.506841 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f864b13b-db9e-42a2-81cd-bfa989374494 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:07.510226 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:07.556343 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e43686-fdb6-4617-9fa2-cb96033195d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:07.557051 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e43686-fdb6-4617-9fa2-cb96033195d9 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:07.557335 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e43686-fdb6-4617-9fa2-cb96033195d9 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:07.557606 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e43686-fdb6-4617-9fa2-cb96033195d9 None admin] PATH_INFO: `/v3/projects/749c6d651fc445cdb75db25a73873386` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:07.558785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e43686-fdb6-4617-9fa2-cb96033195d9 None admin] RBAC: Authorizing `identity:get_project(project_id=749c6d651fc445cdb75db25a73873386)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:07.560311 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e43686-fdb6-4617-9fa2-cb96033195d9 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:07.562536 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 110/207] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:07 2022] GET /identity/v3/projects/749c6d651fc445cdb75db25a73873386 => generated 299 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:07.568560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3c5997-5c68-42a3-98c8-808bd73614e3 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:07.639613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee56934-af43-4843-9e9a-1f999dd99b3a None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:07.640032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee56934-af43-4843-9e9a-1f999dd99b3a None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:07.640607 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee56934-af43-4843-9e9a-1f999dd99b3a None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:07.642029 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 111/208] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:07 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:07.651876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5fc437-dbe1-483e-85bb-68c77ba1b885 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:07.652489 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5fc437-dbe1-483e-85bb-68c77ba1b885 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:07.652951 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5fc437-dbe1-483e-85bb-68c77ba1b885 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:07.683708 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-ff5fc437-dbe1-483e-85bb-68c77ba1b885 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:07.685417 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-ff5fc437-dbe1-483e-85bb-68c77ba1b885 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:07.742852 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3c5997-5c68-42a3-98c8-808bd73614e3 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:07.806543 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 112/209] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:07 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:07.806940 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa769368-2dab-43c6-9740-06d4d8d01b46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:07.807556 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa769368-2dab-43c6-9740-06d4d8d01b46 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:07.807873 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa769368-2dab-43c6-9740-06d4d8d01b46 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:07.808148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa769368-2dab-43c6-9740-06d4d8d01b46 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:07.808895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa769368-2dab-43c6-9740-06d4d8d01b46 None admin] RBAC: Adding query filter params (role.id=868817cab1414b5aa207bc5bdf4f1d9a, scope.project.id=749c6d651fc445cdb75db25a73873386, user.id=ed4c3aa4c7b14a35b57177bf48a0e250) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:07.809206 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa769368-2dab-43c6-9740-06d4d8d01b46 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:07.810643 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa769368-2dab-43c6-9740-06d4d8d01b46 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:07.825178 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 98/210] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:33:07 2022] GET /identity/v3/role_assignments?user.id=ed4c3aa4c7b14a35b57177bf48a0e250&scope.project.id=749c6d651fc445cdb75db25a73873386&role.id=868817cab1414b5aa207bc5bdf4f1d9a => generated 610 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:07.835356 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3d9fe6-5247-43c7-a42d-4de2ef34d6e5 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:07.835721 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3d9fe6-5247-43c7-a42d-4de2ef34d6e5 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:07.835988 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3d9fe6-5247-43c7-a42d-4de2ef34d6e5 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:07.838900 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 113/211] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:07 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:07.840115 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93da4840-4376-4cd2-b3e4-86dcc2028bba None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:07.840627 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93da4840-4376-4cd2-b3e4-86dcc2028bba None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:07.841128 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93da4840-4376-4cd2-b3e4-86dcc2028bba None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:07.842757 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 99/212] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:07 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:33:07.845812 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62cd202-791a-4dce-b08d-f0ea00345471 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:07.846066 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62cd202-791a-4dce-b08d-f0ea00345471 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:07.846336 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62cd202-791a-4dce-b08d-f0ea00345471 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:07.857795 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f60d555-d282-4718-b24d-d06ed850414c None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:07.858106 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f60d555-d282-4718-b24d-d06ed850414c None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:07.858352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f60d555-d282-4718-b24d-d06ed850414c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:07.889750 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-a62cd202-791a-4dce-b08d-f0ea00345471 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:07.891507 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-a62cd202-791a-4dce-b08d-f0ea00345471 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:07.899496 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-5f60d555-d282-4718-b24d-d06ed850414c None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:07.900740 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-5f60d555-d282-4718-b24d-d06ed850414c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:07.962953 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:08.005727 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 100/213] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:07 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:08.024818 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 114/214] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:07 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:08.100408 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f60d555-d282-4718-b24d-d06ed850414c None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:08.214545 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f60d555-d282-4718-b24d-d06ed850414c None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:08.254285 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9567bb7e-d7e0-4839-8fb1-7f96580812fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:08.255207 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9567bb7e-d7e0-4839-8fb1-7f96580812fd None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:08.255496 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9567bb7e-d7e0-4839-8fb1-7f96580812fd None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:08.255802 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9567bb7e-d7e0-4839-8fb1-7f96580812fd None admin] PATH_INFO: `/v3/roles/member` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:08.272377 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9567bb7e-d7e0-4839-8fb1-7f96580812fd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:08.277212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-9567bb7e-d7e0-4839-8fb1-7f96580812fd None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:33:08.277212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:08.277212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:08.277212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:08.277212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:08.277212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:08.277212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:08.277212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:08.277212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:08.277212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:08.277212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:08.277212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:08.277212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:08.277212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:08.277212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:08.277212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:08.278103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:08.278103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:08.278103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:08.278103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:08.278103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:08.278103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:08.278103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:08.278103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:08.278103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:08.278103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:08.278103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:08.278103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:08.278103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:08.278103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:08.278103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:08.278103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:08.278872 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:08.278872 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:08.278872 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:08.278872 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:08.278872 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:08.278872 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:08.278872 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:08.278872 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:08.278872 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:08.278872 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:08.278872 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:33:08.278872 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:08.278872 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 101/215] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:33:08 2022] GET /identity/v3/roles/member => generated 84 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:08.285577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62cd202-791a-4dce-b08d-f0ea00345471 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:08.299176 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9567bb7e-d7e0-4839-8fb1-7f96580812fd None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:08.340235 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:08.357317 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:08.408196 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62cd202-791a-4dce-b08d-f0ea00345471 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:08.417690 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9567bb7e-d7e0-4839-8fb1-7f96580812fd None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:08.444065 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982cee77-fe24-4b26-b7b8-ab64f0d9a7f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:08.444681 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982cee77-fe24-4b26-b7b8-ab64f0d9a7f4 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:08.444961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982cee77-fe24-4b26-b7b8-ab64f0d9a7f4 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:08.445241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982cee77-fe24-4b26-b7b8-ab64f0d9a7f4 None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:08.445915 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982cee77-fe24-4b26-b7b8-ab64f0d9a7f4 None admin] RBAC: Adding query filter params (name=member) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:08.446219 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982cee77-fe24-4b26-b7b8-ab64f0d9a7f4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:08.447598 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982cee77-fe24-4b26-b7b8-ab64f0d9a7f4 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:08.453608 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516e59ff-8db1-4a4e-abbf-41247bad9885 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:08.454247 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516e59ff-8db1-4a4e-abbf-41247bad9885 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:08.454527 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516e59ff-8db1-4a4e-abbf-41247bad9885 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:08.454801 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516e59ff-8db1-4a4e-abbf-41247bad9885 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:08.456321 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 115/216] 10.222.0.102 () {56 vars in 1106 bytes} [Wed Aug 24 15:33:08 2022] GET /identity/v3/roles?name=member => generated 343 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:08.457126 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982cee77-fe24-4b26-b7b8-ab64f0d9a7f4 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:08.469891 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-516e59ff-8db1-4a4e-abbf-41247bad9885 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:08.474575 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-516e59ff-8db1-4a4e-abbf-41247bad9885 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:33:08.474575 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:08.474575 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:08.474575 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:08.474575 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:08.474575 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:08.474575 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:08.474575 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:08.474575 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:08.474575 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:08.474575 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:08.474575 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:08.474575 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:08.474575 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:08.474575 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:08.474575 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:08.475736 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:08.475736 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:08.475736 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:08.475736 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:08.475736 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:08.475736 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:08.475736 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:08.475736 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:08.475736 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:08.475736 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:08.475736 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:08.475736 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:08.475736 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:08.475736 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:08.475736 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:08.475736 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:08.476720 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:08.476720 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:08.476720 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:08.476720 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:08.476720 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:08.476720 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:08.476720 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:08.476720 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:08.476720 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:08.476720 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:08.476720 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:33:08.476720 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:08.476720 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 102/217] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:33:08 2022] GET /identity/v3/roles/admin => generated 83 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:08.477361 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c60d64-8850-4e7e-9eeb-c5464df3054f None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:08.477361 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c60d64-8850-4e7e-9eeb-c5464df3054f None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:08.477474 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c60d64-8850-4e7e-9eeb-c5464df3054f None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:08.478681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 103/218] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:08 2022] GET /identity => generated 271 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:08.479383 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c60d64-8850-4e7e-9eeb-c5464df3054f None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:08.555490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:08.563563 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982cee77-fe24-4b26-b7b8-ab64f0d9a7f4 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:08.586464 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c60d64-8850-4e7e-9eeb-c5464df3054f None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:08.588050 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:08.597968 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3926b919-aa68-4cdb-a810-64bbad61bb02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:08.598598 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3926b919-aa68-4cdb-a810-64bbad61bb02 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:08.598874 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3926b919-aa68-4cdb-a810-64bbad61bb02 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:08.599142 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3926b919-aa68-4cdb-a810-64bbad61bb02 None admin] PATH_INFO: `/v3/domains/default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:08.600249 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3926b919-aa68-4cdb-a810-64bbad61bb02 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:08.601702 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3926b919-aa68-4cdb-a810-64bbad61bb02 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:08.603273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 116/219] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:33:08 2022] GET /identity/v3/domains/default => generated 200 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:08.604057 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3926b919-aa68-4cdb-a810-64bbad61bb02 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:08.625115 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfa130c-5c1d-4a3e-88af-f08a960ab84c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:08.626352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfa130c-5c1d-4a3e-88af-f08a960ab84c None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:08.626645 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfa130c-5c1d-4a3e-88af-f08a960ab84c None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:08.626920 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfa130c-5c1d-4a3e-88af-f08a960ab84c None admin] PATH_INFO: `/v3/users/e2cad07e3619484885c0890917feeea4` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:08.641763 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bfa130c-5c1d-4a3e-88af-f08a960ab84c None admin] RBAC: Authorizing `identity:get_user(user_id=e2cad07e3619484885c0890917feeea4)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:08.643251 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bfa130c-5c1d-4a3e-88af-f08a960ab84c None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:08.644965 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 104/220] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:33:08 2022] GET /identity/v3/users/e2cad07e3619484885c0890917feeea4 => generated 298 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:08.646124 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdb2a8d-96ab-4762-bbc7-1cc0444da353 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:08.646445 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdb2a8d-96ab-4762-bbc7-1cc0444da353 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:08.646746 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdb2a8d-96ab-4762-bbc7-1cc0444da353 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:08.664968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-1fdb2a8d-96ab-4762-bbc7-1cc0444da353 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:08.666160 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-1fdb2a8d-96ab-4762-bbc7-1cc0444da353 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:08.716524 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3926b919-aa68-4cdb-a810-64bbad61bb02 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:08.744206 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 105/221] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:08 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:08.745004 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fdb2a8d-96ab-4762-bbc7-1cc0444da353 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:08.753960 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc83cca9-117a-4561-8797-f5f2dc3d2dd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:08.754585 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc83cca9-117a-4561-8797-f5f2dc3d2dd3 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:08.754929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc83cca9-117a-4561-8797-f5f2dc3d2dd3 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:08.755212 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc83cca9-117a-4561-8797-f5f2dc3d2dd3 None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:08.755925 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc83cca9-117a-4561-8797-f5f2dc3d2dd3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:08.756233 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc83cca9-117a-4561-8797-f5f2dc3d2dd3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:08.757635 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc83cca9-117a-4561-8797-f5f2dc3d2dd3 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:08.766913 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 117/222] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:33:08 2022] GET /identity/v3/roles?name=admin => generated 341 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:08.767779 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc83cca9-117a-4561-8797-f5f2dc3d2dd3 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:08.801547 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:08.865489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:08.877277 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fdb2a8d-96ab-4762-bbc7-1cc0444da353 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:08.893791 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc83cca9-117a-4561-8797-f5f2dc3d2dd3 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:08.924073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249d85b5-3c15-4d2f-bf44-32738b9e463c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:08.925344 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249d85b5-3c15-4d2f-bf44-32738b9e463c None admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:08.925666 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249d85b5-3c15-4d2f-bf44-32738b9e463c None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:08.926130 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249d85b5-3c15-4d2f-bf44-32738b9e463c None admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:08.927235 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249d85b5-3c15-4d2f-bf44-32738b9e463c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:08.930078 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249d85b5-3c15-4d2f-bf44-32738b9e463c None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:08.936331 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6623765-63bc-4c46-bef8-91f839e9462b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:08.938341 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6623765-63bc-4c46-bef8-91f839e9462b None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:08.938871 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6623765-63bc-4c46-bef8-91f839e9462b None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:08.939285 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6623765-63bc-4c46-bef8-91f839e9462b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:08.964128 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 106/223] 10.222.0.102 () {60 vars in 1137 bytes} [Wed Aug 24 15:33:08 2022] POST /identity/v3/users => generated 298 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:33:08.964792 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249d85b5-3c15-4d2f-bf44-32738b9e463c None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:09.049589 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6623765-63bc-4c46-bef8-91f839e9462b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:09.051329 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6623765-63bc-4c46-bef8-91f839e9462b None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:09.083976 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:09.101126 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249d85b5-3c15-4d2f-bf44-32738b9e463c None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:09.146315 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a85903-fde5-42e1-8edd-61021a657667 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:09.146957 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a85903-fde5-42e1-8edd-61021a657667 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:09.147704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a85903-fde5-42e1-8edd-61021a657667 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:09.147704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a85903-fde5-42e1-8edd-61021a657667 None admin] PATH_INFO: `/v3/users/bb05bf01d9d24a5296517f8bde87462b` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:09.148708 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a85903-fde5-42e1-8edd-61021a657667 None admin] RBAC: Authorizing `identity:get_user(user_id=bb05bf01d9d24a5296517f8bde87462b)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:09.150192 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a85903-fde5-42e1-8edd-61021a657667 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:09.152968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 107/224] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:33:08 2022] GET /identity/v3/users/bb05bf01d9d24a5296517f8bde87462b => generated 284 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:09.154694 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a85903-fde5-42e1-8edd-61021a657667 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:09.175708 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:09.186684 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 118/225] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:33:08 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 419 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:33:09.188403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6623765-63bc-4c46-bef8-91f839e9462b None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:09.292819 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a85903-fde5-42e1-8edd-61021a657667 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:09.320329 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6623765-63bc-4c46-bef8-91f839e9462b None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:09.350145 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583d6b54-e653-496e-80ea-e6bfed72e11b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:09.350781 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583d6b54-e653-496e-80ea-e6bfed72e11b None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:09.351182 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583d6b54-e653-496e-80ea-e6bfed72e11b None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:09.351455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583d6b54-e653-496e-80ea-e6bfed72e11b None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:09.363518 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:09.367481 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583d6b54-e653-496e-80ea-e6bfed72e11b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:09.375681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-583d6b54-e653-496e-80ea-e6bfed72e11b None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 15:33:09.375681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:09.375681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:09.375681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:09.375681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:09.375681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:09.375681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:09.375681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:09.375681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:09.375681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:09.375681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:09.375681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:09.375681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:09.375681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:09.375681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:09.375681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:09.376700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:09.376700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:09.376700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:09.376700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:09.376700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:09.376700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:09.376700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:09.376700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:09.376700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:09.376700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:09.376700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:09.376700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:09.376700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:09.376700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:09.376700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:09.376700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:09.377523 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:09.377523 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:09.377523 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:09.377523 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:09.377523 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:09.377523 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:09.377523 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:09.377523 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:09.377523 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:09.377523 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:09.377523 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 15:33:09.377523 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:09.377523 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 108/226] 10.222.0.102 () {56 vars in 1119 bytes} [Wed Aug 24 15:33:09 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:09.377523 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583d6b54-e653-496e-80ea-e6bfed72e11b None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:09.385342 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be81bd4c-8642-4996-8fd0-0419ab99067b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:09.385998 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be81bd4c-8642-4996-8fd0-0419ab99067b None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:09.386276 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be81bd4c-8642-4996-8fd0-0419ab99067b None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:09.386557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be81bd4c-8642-4996-8fd0-0419ab99067b None admin] PATH_INFO: `/v3/projects/86a6b464bc2d4f72895f92c759e26a36` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:09.387748 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be81bd4c-8642-4996-8fd0-0419ab99067b None admin] RBAC: Authorizing `identity:get_project(project_id=86a6b464bc2d4f72895f92c759e26a36)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:09.389884 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be81bd4c-8642-4996-8fd0-0419ab99067b None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:09.392592 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 119/227] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:09 2022] GET /identity/v3/projects/86a6b464bc2d4f72895f92c759e26a36 => generated 303 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:09.393941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be81bd4c-8642-4996-8fd0-0419ab99067b None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:09.484278 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:09.520767 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583d6b54-e653-496e-80ea-e6bfed72e11b None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:09.541687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be81bd4c-8642-4996-8fd0-0419ab99067b None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:09.565119 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8be0b48-0034-452d-83e8-d99d5b2ce171 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:09.566160 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8be0b48-0034-452d-83e8-d99d5b2ce171 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:09.566440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8be0b48-0034-452d-83e8-d99d5b2ce171 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:09.566712 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8be0b48-0034-452d-83e8-d99d5b2ce171 None admin] PATH_INFO: `/v3/projects/86a6b464bc2d4f72895f92c759e26a36` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:09.567891 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8be0b48-0034-452d-83e8-d99d5b2ce171 None admin] RBAC: Authorizing `identity:get_project(project_id=86a6b464bc2d4f72895f92c759e26a36)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:09.569944 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8be0b48-0034-452d-83e8-d99d5b2ce171 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:09.575788 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 109/228] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:09 2022] GET /identity/v3/projects/86a6b464bc2d4f72895f92c759e26a36 => generated 303 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:09.576553 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8be0b48-0034-452d-83e8-d99d5b2ce171 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:09.589934 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da92ebb9-504d-4804-a1c9-901dd0de7a0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:09.590581 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da92ebb9-504d-4804-a1c9-901dd0de7a0c None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:09.590833 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da92ebb9-504d-4804-a1c9-901dd0de7a0c None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:09.591108 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da92ebb9-504d-4804-a1c9-901dd0de7a0c None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:09.591840 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da92ebb9-504d-4804-a1c9-901dd0de7a0c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:09.592219 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da92ebb9-504d-4804-a1c9-901dd0de7a0c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:09.594685 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da92ebb9-504d-4804-a1c9-901dd0de7a0c None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:09.604045 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 120/229] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:33:09 2022] GET /identity/v3/roles?name=anotherrole => generated 336 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:09.604877 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da92ebb9-504d-4804-a1c9-901dd0de7a0c None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:09.672554 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:09.721937 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8be0b48-0034-452d-83e8-d99d5b2ce171 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:09.733901 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da92ebb9-504d-4804-a1c9-901dd0de7a0c None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:09.765898 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efa5c04-77f7-4ceb-b5c5-05babe77ff55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:09.766831 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efa5c04-77f7-4ceb-b5c5-05babe77ff55 None admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:09.767109 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efa5c04-77f7-4ceb-b5c5-05babe77ff55 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:09.767382 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efa5c04-77f7-4ceb-b5c5-05babe77ff55 None admin] PATH_INFO: `/v3/projects/86a6b464bc2d4f72895f92c759e26a36/users/bb05bf01d9d24a5296517f8bde87462b/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:09.770199 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3efa5c04-77f7-4ceb-b5c5-05babe77ff55 None admin] RBAC: Authorizing `identity:create_grant(project_id=86a6b464bc2d4f72895f92c759e26a36, user_id=bb05bf01d9d24a5296517f8bde87462b, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:09.772143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3efa5c04-77f7-4ceb-b5c5-05babe77ff55 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:09.784654 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b061fc69-1d0e-4991-89e5-222abd9f07c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:09.785641 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b061fc69-1d0e-4991-89e5-222abd9f07c8 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:09.786183 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b061fc69-1d0e-4991-89e5-222abd9f07c8 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:09.786813 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b061fc69-1d0e-4991-89e5-222abd9f07c8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:09.788050 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b061fc69-1d0e-4991-89e5-222abd9f07c8 None admin] RBAC: Adding query filter params (role.id=b5d2ed3e20ab47eeb741b4f43ab2eebb, scope.project.id=86a6b464bc2d4f72895f92c759e26a36, user.id=e2cad07e3619484885c0890917feeea4) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:09.788565 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b061fc69-1d0e-4991-89e5-222abd9f07c8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:09.790223 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b061fc69-1d0e-4991-89e5-222abd9f07c8 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:09.796334 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 110/230] 10.222.0.102 () {58 vars in 1444 bytes} [Wed Aug 24 15:33:09 2022] PUT /identity/v3/projects/86a6b464bc2d4f72895f92c759e26a36/users/bb05bf01d9d24a5296517f8bde87462b/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:33:09.796334 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efa5c04-77f7-4ceb-b5c5-05babe77ff55 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:09.797415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:09.810612 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 121/231] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:33:09 2022] GET /identity/v3/role_assignments?user.id=e2cad07e3619484885c0890917feeea4&scope.project.id=86a6b464bc2d4f72895f92c759e26a36&role.id=b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 260 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:09.975937 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efa5c04-77f7-4ceb-b5c5-05babe77ff55 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:10.020190 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57bbf43c-5e42-479b-90f9-43c62445c599 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:10.020826 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57bbf43c-5e42-479b-90f9-43c62445c599 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:10.021104 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57bbf43c-5e42-479b-90f9-43c62445c599 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:10.021373 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57bbf43c-5e42-479b-90f9-43c62445c599 None admin] PATH_INFO: `/v3/users/bb05bf01d9d24a5296517f8bde87462b` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:10.022498 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57bbf43c-5e42-479b-90f9-43c62445c599 None admin] RBAC: Authorizing `identity:get_user(user_id=bb05bf01d9d24a5296517f8bde87462b)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:10.023969 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57bbf43c-5e42-479b-90f9-43c62445c599 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:10.025834 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 111/232] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:33:09 2022] GET /identity/v3/users/bb05bf01d9d24a5296517f8bde87462b => generated 284 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:10.037736 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b061fc69-1d0e-4991-89e5-222abd9f07c8 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:10.175362 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b061fc69-1d0e-4991-89e5-222abd9f07c8 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:10.195655 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:10.219895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53696a8-a528-4250-afbd-f052a78b96ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:10.220976 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53696a8-a528-4250-afbd-f052a78b96ad None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:10.221482 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53696a8-a528-4250-afbd-f052a78b96ad None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:10.221953 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53696a8-a528-4250-afbd-f052a78b96ad None admin] PATH_INFO: `/v3/projects/86a6b464bc2d4f72895f92c759e26a36` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:10.223319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53696a8-a528-4250-afbd-f052a78b96ad None admin] RBAC: Authorizing `identity:get_project(project_id=86a6b464bc2d4f72895f92c759e26a36)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:10.225101 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53696a8-a528-4250-afbd-f052a78b96ad None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:10.227056 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 122/233] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:10 2022] GET /identity/v3/projects/86a6b464bc2d4f72895f92c759e26a36 => generated 303 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:10.234514 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57bbf43c-5e42-479b-90f9-43c62445c599 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:10.342247 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:10.376328 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57bbf43c-5e42-479b-90f9-43c62445c599 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:10.412415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868487bc-00c6-44f9-86fa-90e5bb8ddc7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:10.413099 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868487bc-00c6-44f9-86fa-90e5bb8ddc7b None admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:10.413377 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868487bc-00c6-44f9-86fa-90e5bb8ddc7b None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:10.413648 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868487bc-00c6-44f9-86fa-90e5bb8ddc7b None admin] PATH_INFO: `/v3/projects/86a6b464bc2d4f72895f92c759e26a36/users/bb05bf01d9d24a5296517f8bde87462b/roles/b8429ebd9c2a4cbbbbf30ad56ce23bfa` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:10.416010 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868487bc-00c6-44f9-86fa-90e5bb8ddc7b None admin] RBAC: Authorizing `identity:create_grant(project_id=86a6b464bc2d4f72895f92c759e26a36, user_id=bb05bf01d9d24a5296517f8bde87462b, role_id=b8429ebd9c2a4cbbbbf30ad56ce23bfa)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:10.417493 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868487bc-00c6-44f9-86fa-90e5bb8ddc7b None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:10.438874 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 112/234] 10.222.0.102 () {58 vars in 1444 bytes} [Wed Aug 24 15:33:10 2022] PUT /identity/v3/projects/86a6b464bc2d4f72895f92c759e26a36/users/bb05bf01d9d24a5296517f8bde87462b/roles/b8429ebd9c2a4cbbbbf30ad56ce23bfa => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:33:11.451048 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f08dfdb-f109-4ee8-ba3d-8aebf6af66a3 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:11.452090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f08dfdb-f109-4ee8-ba3d-8aebf6af66a3 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:11.453276 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f08dfdb-f109-4ee8-ba3d-8aebf6af66a3 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:11.454845 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 123/235] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:11 2022] GET /identity => generated 271 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:11.460160 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e030e8-18df-4ddf-b5b6-09f0a105d4f6 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:11.461005 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e030e8-18df-4ddf-b5b6-09f0a105d4f6 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:11.461478 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e030e8-18df-4ddf-b5b6-09f0a105d4f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:11.474107 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3e626d-9be7-4b38-8a34-e076c0932608 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:11.474654 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3e626d-9be7-4b38-8a34-e076c0932608 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:11.475094 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3e626d-9be7-4b38-8a34-e076c0932608 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:11.476872 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 124/236] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:11 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:11.482522 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5353d980-7e73-4175-8b73-6b0c5c721c95 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:11.483059 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5353d980-7e73-4175-8b73-6b0c5c721c95 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:11.483551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5353d980-7e73-4175-8b73-6b0c5c721c95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:11.488345 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-d9e030e8-18df-4ddf-b5b6-09f0a105d4f6 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:11.490297 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-d9e030e8-18df-4ddf-b5b6-09f0a105d4f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:11.510326 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-5353d980-7e73-4175-8b73-6b0c5c721c95 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:11.514236 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-5353d980-7e73-4175-8b73-6b0c5c721c95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:11.628042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 113/237] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:11 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 15:33:11.653951 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 125/238] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:11 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:11.947666 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e030e8-18df-4ddf-b5b6-09f0a105d4f6 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:12.035048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:12.050001 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5353d980-7e73-4175-8b73-6b0c5c721c95 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:12.135506 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e030e8-18df-4ddf-b5b6-09f0a105d4f6 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:12.209908 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5a8b76-a604-4f28-b858-53290ccea4d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:12.210546 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5a8b76-a604-4f28-b858-53290ccea4d2 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:12.210829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5a8b76-a604-4f28-b858-53290ccea4d2 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:12.211101 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5a8b76-a604-4f28-b858-53290ccea4d2 None admin] PATH_INFO: `/v3/domains/default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:12.212323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f5a8b76-a604-4f28-b858-53290ccea4d2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:12.213788 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f5a8b76-a604-4f28-b858-53290ccea4d2 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:12.215378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 114/239] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:33:11 2022] GET /identity/v3/domains/default => generated 200 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:12.221821 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5a8b76-a604-4f28-b858-53290ccea4d2 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:12.261328 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5353d980-7e73-4175-8b73-6b0c5c721c95 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:12.303063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:12.326326 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34208d19-07be-4618-aea6-eb2d98266d20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:12.327531 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34208d19-07be-4618-aea6-eb2d98266d20 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:12.328010 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34208d19-07be-4618-aea6-eb2d98266d20 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:12.328938 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34208d19-07be-4618-aea6-eb2d98266d20 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:12.342642 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34208d19-07be-4618-aea6-eb2d98266d20 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:12.349571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-34208d19-07be-4618-aea6-eb2d98266d20 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 15:33:12.349571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:12.349571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:12.349571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:12.349571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:12.349571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:12.349571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:12.349571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:12.349571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:12.349571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:12.349571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:12.349571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:12.349571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:12.349571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:12.349571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:12.349571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:12.350673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:12.350673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:12.350673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:12.350673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:12.350673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:12.350673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:12.350673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:12.350673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:12.350673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:12.350673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:12.350673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:12.350673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:12.350673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:12.350673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:12.350673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:12.350673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:12.351674 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:12.351674 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:12.351674 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:12.351674 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:12.351674 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:12.351674 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:12.351674 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:12.351674 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:12.351674 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:12.351674 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:12.351674 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 15:33:12.351674 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:12.352977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 126/240] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:33:12 2022] GET /identity/v3/roles/reader => generated 84 bytes in 303 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:33:12.356839 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549f62bc-01a2-4e9d-9885-805d7ed1c499 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:12.357382 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549f62bc-01a2-4e9d-9885-805d7ed1c499 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:12.357865 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549f62bc-01a2-4e9d-9885-805d7ed1c499 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:12.365127 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 127/241] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:12 2022] GET /identity => generated 271 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:12.367495 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549f62bc-01a2-4e9d-9885-805d7ed1c499 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:12.393574 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5a8b76-a604-4f28-b858-53290ccea4d2 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:12.424242 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:12.433623 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3049e670-11f0-474e-93f6-c5dd4eb10f0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:12.434539 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3049e670-11f0-474e-93f6-c5dd4eb10f0e None admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:12.435031 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3049e670-11f0-474e-93f6-c5dd4eb10f0e None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:12.435503 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3049e670-11f0-474e-93f6-c5dd4eb10f0e None admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:12.436410 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3049e670-11f0-474e-93f6-c5dd4eb10f0e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:12.438735 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3049e670-11f0-474e-93f6-c5dd4eb10f0e None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:12.464201 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 115/242] 10.222.0.102 () {60 vars in 1137 bytes} [Wed Aug 24 15:33:12 2022] POST /identity/v3/users => generated 294 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:33:12.466352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e5b84b-a195-4a1f-9937-92a3423e9c96 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:12.466847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e5b84b-a195-4a1f-9937-92a3423e9c96 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:12.467322 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e5b84b-a195-4a1f-9937-92a3423e9c96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:12.494180 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-86e5b84b-a195-4a1f-9937-92a3423e9c96 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:12.494672 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549f62bc-01a2-4e9d-9885-805d7ed1c499 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:12.496797 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-86e5b84b-a195-4a1f-9937-92a3423e9c96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:12.545166 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-accacee9-4bd3-4cf0-b7bb-e15df4e3a234 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:12.546293 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-accacee9-4bd3-4cf0-b7bb-e15df4e3a234 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:12.548096 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-accacee9-4bd3-4cf0-b7bb-e15df4e3a234 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:12.548382 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-accacee9-4bd3-4cf0-b7bb-e15df4e3a234 None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:12.552162 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-accacee9-4bd3-4cf0-b7bb-e15df4e3a234 None admin] RBAC: Adding query filter params (name=reader) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:12.552729 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-accacee9-4bd3-4cf0-b7bb-e15df4e3a234 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:12.555092 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-accacee9-4bd3-4cf0-b7bb-e15df4e3a234 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:12.573120 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 128/243] 10.222.0.102 () {56 vars in 1106 bytes} [Wed Aug 24 15:33:12 2022] GET /identity/v3/roles?name=reader => generated 343 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:12.573618 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd8f176-e6eb-4bb8-ad47-ca371b473608 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:12.573744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd8f176-e6eb-4bb8-ad47-ca371b473608 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:12.573962 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd8f176-e6eb-4bb8-ad47-ca371b473608 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:12.577901 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 129/244] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:12 2022] GET /identity => generated 271 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:12.580470 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd8f176-e6eb-4bb8-ad47-ca371b473608 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:12.607742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 116/245] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:12 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:12.610845 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64394764-cb52-4dd7-b4b3-7ad99603f5a6 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:12.611357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64394764-cb52-4dd7-b4b3-7ad99603f5a6 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:12.612110 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64394764-cb52-4dd7-b4b3-7ad99603f5a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:12.638223 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-64394764-cb52-4dd7-b4b3-7ad99603f5a6 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:12.639410 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-64394764-cb52-4dd7-b4b3-7ad99603f5a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:12.649835 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:12.734093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 117/246] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:12 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:12.750581 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd8f176-e6eb-4bb8-ad47-ca371b473608 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:12.787202 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de66bf20-9a7c-4dfe-abcd-9438c18e8415 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:12.788212 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de66bf20-9a7c-4dfe-abcd-9438c18e8415 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:12.788712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de66bf20-9a7c-4dfe-abcd-9438c18e8415 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:12.789170 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de66bf20-9a7c-4dfe-abcd-9438c18e8415 None admin] PATH_INFO: `/v3/users/b59f7c2f0d1c42a6816994bf24188271` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:12.805029 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de66bf20-9a7c-4dfe-abcd-9438c18e8415 None admin] RBAC: Authorizing `identity:get_user(user_id=b59f7c2f0d1c42a6816994bf24188271)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:12.806934 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de66bf20-9a7c-4dfe-abcd-9438c18e8415 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:12.808857 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 130/247] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:33:12 2022] GET /identity/v3/users/b59f7c2f0d1c42a6816994bf24188271 => generated 298 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:12.817668 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64394764-cb52-4dd7-b4b3-7ad99603f5a6 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:12.892126 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:12.963871 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de66bf20-9a7c-4dfe-abcd-9438c18e8415 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:12.973751 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64394764-cb52-4dd7-b4b3-7ad99603f5a6 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:13.025999 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23b8447-c2f3-46e6-8935-ecf07a9ec863 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:13.026584 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23b8447-c2f3-46e6-8935-ecf07a9ec863 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:13.027517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23b8447-c2f3-46e6-8935-ecf07a9ec863 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:13.027517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23b8447-c2f3-46e6-8935-ecf07a9ec863 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:13.107797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:13.113897 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de66bf20-9a7c-4dfe-abcd-9438c18e8415 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:13.128138 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d23b8447-c2f3-46e6-8935-ecf07a9ec863 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:13.129628 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d23b8447-c2f3-46e6-8935-ecf07a9ec863 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:13.158556 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54bd03c8-ed47-431f-8a28-6f9f74c79d45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:13.159513 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bd03c8-ed47-431f-8a28-6f9f74c79d45 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:13.159826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bd03c8-ed47-431f-8a28-6f9f74c79d45 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:13.160098 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bd03c8-ed47-431f-8a28-6f9f74c79d45 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:13.176522 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54bd03c8-ed47-431f-8a28-6f9f74c79d45 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:13.184537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-54bd03c8-ed47-431f-8a28-6f9f74c79d45 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:33:13.184537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:13.184537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:13.184537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:13.184537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:13.184537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:13.184537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:13.184537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:13.184537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:13.184537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:13.184537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:13.184537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:13.184537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:13.184537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:13.184537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:13.184537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:13.185454 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:13.185454 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:13.185454 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:13.185454 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:13.185454 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:13.185454 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:13.185454 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:13.185454 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:13.185454 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:13.185454 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:13.185454 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:13.185454 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:13.185454 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:13.185454 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:13.185454 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:13.185454 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:13.186220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:13.186220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:13.186220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:13.186220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:13.186220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:13.186220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:13.186220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:13.186220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:13.186220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:13.186220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:13.186220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:33:13.186220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:13.187970 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:13.191197 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 131/248] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:33:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:33:13.193077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fc15c8-7d40-4c23-9e12-9ce96810e6b9 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:13.193510 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fc15c8-7d40-4c23-9e12-9ce96810e6b9 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:13.194243 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fc15c8-7d40-4c23-9e12-9ce96810e6b9 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:13.196083 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 132/249] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:13 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:13.196989 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fc15c8-7d40-4c23-9e12-9ce96810e6b9 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:13.264584 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 118/250] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:33:12 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 443 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 24 15:33:13.264584 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23b8447-c2f3-46e6-8935-ecf07a9ec863 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:13.332278 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fc15c8-7d40-4c23-9e12-9ce96810e6b9 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:13.368872 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5402d3c4-13f0-487a-a717-74c30f25fa15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:13.369690 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5402d3c4-13f0-487a-a717-74c30f25fa15 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:13.370059 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5402d3c4-13f0-487a-a717-74c30f25fa15 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:13.370426 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5402d3c4-13f0-487a-a717-74c30f25fa15 None admin] PATH_INFO: `/v3/roles/member` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:13.381282 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5402d3c4-13f0-487a-a717-74c30f25fa15 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:13.381573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23b8447-c2f3-46e6-8935-ecf07a9ec863 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:13.386516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-5402d3c4-13f0-487a-a717-74c30f25fa15 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:33:13.386516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:13.386516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:13.386516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:13.386516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:13.386516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:13.386516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:13.386516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:13.386516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:13.386516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:13.386516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:13.386516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:13.386516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:13.386516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:13.386516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:13.386516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:13.387506 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:13.387506 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:13.387506 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:13.387506 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:13.387506 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:13.387506 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:13.387506 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:13.387506 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:13.387506 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:13.387506 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:13.387506 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:13.387506 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:13.387506 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:13.387506 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:13.387506 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:13.387506 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:13.388322 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:13.388322 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:13.388322 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:13.388322 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:13.388322 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:13.388322 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:13.388322 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:13.388322 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:13.388322 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:13.388322 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:13.388322 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:33:13.388322 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:13.393359 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 133/251] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:33:13 2022] GET /identity/v3/roles/member => generated 84 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:13.394813 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd99ca5-c840-44f5-a3b1-3ee76f0475c0 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:13.395162 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd99ca5-c840-44f5-a3b1-3ee76f0475c0 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:13.395515 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd99ca5-c840-44f5-a3b1-3ee76f0475c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:13.415288 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-1fd99ca5-c840-44f5-a3b1-3ee76f0475c0 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:13.416618 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-1fd99ca5-c840-44f5-a3b1-3ee76f0475c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:13.428864 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:13.429252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b49748-d166-418a-8c42-24eb039d366e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:13.430763 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b49748-d166-418a-8c42-24eb039d366e None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:13.431332 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b49748-d166-418a-8c42-24eb039d366e None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:13.432073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b49748-d166-418a-8c42-24eb039d366e None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:13.432929 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b49748-d166-418a-8c42-24eb039d366e None admin] RBAC: Adding query filter params (name=admin) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:13.433896 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b49748-d166-418a-8c42-24eb039d366e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:13.435987 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b49748-d166-418a-8c42-24eb039d366e None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:13.446364 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 119/252] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:33:13 2022] GET /identity/v3/roles?name=admin => generated 341 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:13.447575 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b49748-d166-418a-8c42-24eb039d366e None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:13.458946 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:13.496042 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 134/253] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:13 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:13.497212 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd99ca5-c840-44f5-a3b1-3ee76f0475c0 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:13.564781 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b49748-d166-418a-8c42-24eb039d366e None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:13.610673 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e40b055-104d-4c47-8a28-56819524f7f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:13.619928 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e40b055-104d-4c47-8a28-56819524f7f6 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:13.620538 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e40b055-104d-4c47-8a28-56819524f7f6 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:13.621019 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e40b055-104d-4c47-8a28-56819524f7f6 None admin] PATH_INFO: `/v3/projects/86a6b464bc2d4f72895f92c759e26a36` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:13.622596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e40b055-104d-4c47-8a28-56819524f7f6 None admin] RBAC: Authorizing `identity:get_project(project_id=86a6b464bc2d4f72895f92c759e26a36)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:13.624919 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e40b055-104d-4c47-8a28-56819524f7f6 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:13.627550 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 120/254] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:13 2022] GET /identity/v3/projects/86a6b464bc2d4f72895f92c759e26a36 => generated 303 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:13.628547 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e40b055-104d-4c47-8a28-56819524f7f6 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:13.647661 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd99ca5-c840-44f5-a3b1-3ee76f0475c0 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:13.682638 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df73618a-63e8-4210-bf99-9e3ec55877c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:13.683248 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df73618a-63e8-4210-bf99-9e3ec55877c0 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:13.683533 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df73618a-63e8-4210-bf99-9e3ec55877c0 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:13.684004 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df73618a-63e8-4210-bf99-9e3ec55877c0 None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:13.684687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df73618a-63e8-4210-bf99-9e3ec55877c0 None admin] RBAC: Adding query filter params (name=member) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:13.684986 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df73618a-63e8-4210-bf99-9e3ec55877c0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:13.686379 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df73618a-63e8-4210-bf99-9e3ec55877c0 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:13.709014 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 135/255] 10.222.0.102 () {56 vars in 1106 bytes} [Wed Aug 24 15:33:13 2022] GET /identity/v3/roles?name=member => generated 343 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:13.709870 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df73618a-63e8-4210-bf99-9e3ec55877c0 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:13.734630 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:13.784049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:13.797587 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e40b055-104d-4c47-8a28-56819524f7f6 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:13.858282 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99a7515-bc58-4d00-944b-699a29d394b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:13.859996 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99a7515-bc58-4d00-944b-699a29d394b2 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:13.860276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99a7515-bc58-4d00-944b-699a29d394b2 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:13.860546 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99a7515-bc58-4d00-944b-699a29d394b2 None admin] PATH_INFO: `/v3/users/bb05bf01d9d24a5296517f8bde87462b` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:13.864482 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c99a7515-bc58-4d00-944b-699a29d394b2 None admin] RBAC: Authorizing `identity:get_user(user_id=bb05bf01d9d24a5296517f8bde87462b)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:13.865931 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c99a7515-bc58-4d00-944b-699a29d394b2 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:13.867785 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 121/256] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:33:13 2022] GET /identity/v3/users/bb05bf01d9d24a5296517f8bde87462b => generated 284 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:13.869104 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99a7515-bc58-4d00-944b-699a29d394b2 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:13.960515 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df73618a-63e8-4210-bf99-9e3ec55877c0 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:14.006949 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aceedb9-c109-4d1b-86e9-0083f9dbf619 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:14.008159 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aceedb9-c109-4d1b-86e9-0083f9dbf619 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:14.008995 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aceedb9-c109-4d1b-86e9-0083f9dbf619 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:14.009711 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aceedb9-c109-4d1b-86e9-0083f9dbf619 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:14.010870 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aceedb9-c109-4d1b-86e9-0083f9dbf619 None admin] RBAC: Adding query filter params (role.id=868817cab1414b5aa207bc5bdf4f1d9a, scope.project.id=86a6b464bc2d4f72895f92c759e26a36, user.id=b59f7c2f0d1c42a6816994bf24188271) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:14.011389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aceedb9-c109-4d1b-86e9-0083f9dbf619 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:14.013642 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aceedb9-c109-4d1b-86e9-0083f9dbf619 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:14.018480 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99a7515-bc58-4d00-944b-699a29d394b2 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:14.028280 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 136/257] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:33:13 2022] GET /identity/v3/role_assignments?user.id=b59f7c2f0d1c42a6816994bf24188271&scope.project.id=86a6b464bc2d4f72895f92c759e26a36&role.id=868817cab1414b5aa207bc5bdf4f1d9a => generated 260 bytes in 313 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:14.029171 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aceedb9-c109-4d1b-86e9-0083f9dbf619 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:14.100547 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c67dc11-c406-45df-ab77-d5354b977fef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:14.101125 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c67dc11-c406-45df-ab77-d5354b977fef None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:14.101403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c67dc11-c406-45df-ab77-d5354b977fef None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:14.101675 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c67dc11-c406-45df-ab77-d5354b977fef None admin] PATH_INFO: `/v3/users/e2cad07e3619484885c0890917feeea4` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:14.103854 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c67dc11-c406-45df-ab77-d5354b977fef None admin] RBAC: Authorizing `identity:get_user(user_id=e2cad07e3619484885c0890917feeea4)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:14.105377 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c67dc11-c406-45df-ab77-d5354b977fef None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:14.107128 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 122/258] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:33:13 2022] GET /identity/v3/users/e2cad07e3619484885c0890917feeea4 => generated 298 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:14.107958 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c67dc11-c406-45df-ab77-d5354b977fef None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:14.135087 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:14.172177 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:14.208367 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aceedb9-c109-4d1b-86e9-0083f9dbf619 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:14.254176 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a110d1-630a-45c7-9447-82c2f6967e81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:14.254934 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a110d1-630a-45c7-9447-82c2f6967e81 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:14.255218 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a110d1-630a-45c7-9447-82c2f6967e81 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:14.255515 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a110d1-630a-45c7-9447-82c2f6967e81 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:14.332939 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c67dc11-c406-45df-ab77-d5354b977fef None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:14.381511 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9953d38-bb20-42a6-97a4-7152055114e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:14.382162 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9953d38-bb20-42a6-97a4-7152055114e9 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:14.382443 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9953d38-bb20-42a6-97a4-7152055114e9 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:14.382716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9953d38-bb20-42a6-97a4-7152055114e9 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:14.402110 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a110d1-630a-45c7-9447-82c2f6967e81 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:14.403622 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a110d1-630a-45c7-9447-82c2f6967e81 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:14.409138 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9953d38-bb20-42a6-97a4-7152055114e9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:14.419994 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-d9953d38-bb20-42a6-97a4-7152055114e9 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 15:33:14.419994 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:14.419994 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:14.419994 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:14.419994 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:14.419994 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:14.419994 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:14.419994 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:14.419994 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:14.419994 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:14.419994 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:14.419994 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:14.419994 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:14.419994 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:14.419994 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:14.419994 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:14.421012 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:14.421012 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:14.421012 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:14.421012 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:14.421012 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:14.421012 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:14.421012 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:14.421012 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:14.421012 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:14.421012 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:14.421012 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:14.421012 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:14.421012 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:14.421012 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:14.421012 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:14.421012 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:14.422401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:14.422401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:14.422401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:14.422401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:14.422401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:14.422401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:14.422401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:14.422401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:14.422401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:14.422401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:14.422401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 15:33:14.422401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:14.431709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 123/259] 10.222.0.102 () {56 vars in 1119 bytes} [Wed Aug 24 15:33:14 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 324 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:14.433024 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9953d38-bb20-42a6-97a4-7152055114e9 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:14.556476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:14.580042 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:14.600854 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9953d38-bb20-42a6-97a4-7152055114e9 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:14.636252 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 137/260] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:33:14 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 608 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:33:14.637054 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a110d1-630a-45c7-9447-82c2f6967e81 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:14.668516 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73a83ff-dc36-4dfa-b4a9-23aaf6c33b9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:14.669159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73a83ff-dc36-4dfa-b4a9-23aaf6c33b9e None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:14.669441 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73a83ff-dc36-4dfa-b4a9-23aaf6c33b9e None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:14.669710 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73a83ff-dc36-4dfa-b4a9-23aaf6c33b9e None admin] PATH_INFO: `/v3/projects/86a6b464bc2d4f72895f92c759e26a36` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:14.672540 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73a83ff-dc36-4dfa-b4a9-23aaf6c33b9e None admin] RBAC: Authorizing `identity:get_project(project_id=86a6b464bc2d4f72895f92c759e26a36)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:14.674973 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73a83ff-dc36-4dfa-b4a9-23aaf6c33b9e None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:14.676714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 124/261] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:14 2022] GET /identity/v3/projects/86a6b464bc2d4f72895f92c759e26a36 => generated 303 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:14.677487 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73a83ff-dc36-4dfa-b4a9-23aaf6c33b9e None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:14.809679 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a110d1-630a-45c7-9447-82c2f6967e81 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:14.887017 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c974843-1c59-4552-8193-2e81519d1a2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:14.887568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c974843-1c59-4552-8193-2e81519d1a2b None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:14.887873 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c974843-1c59-4552-8193-2e81519d1a2b None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:14.888151 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c974843-1c59-4552-8193-2e81519d1a2b None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:14.888835 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c974843-1c59-4552-8193-2e81519d1a2b None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:14.889140 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c974843-1c59-4552-8193-2e81519d1a2b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:14.892782 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c974843-1c59-4552-8193-2e81519d1a2b None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:14.898641 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73a83ff-dc36-4dfa-b4a9-23aaf6c33b9e None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:14.910836 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 138/262] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:33:14 2022] GET /identity/v3/roles?name=anotherrole => generated 336 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:14.911676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c974843-1c59-4552-8193-2e81519d1a2b None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:14.944921 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0b8b61-a5e8-47f4-ba29-0ccb51f4c553 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:14.945565 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0b8b61-a5e8-47f4-ba29-0ccb51f4c553 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:14.945848 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0b8b61-a5e8-47f4-ba29-0ccb51f4c553 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:14.946123 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0b8b61-a5e8-47f4-ba29-0ccb51f4c553 None admin] PATH_INFO: `/v3/projects/86a6b464bc2d4f72895f92c759e26a36` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:14.947756 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0b8b61-a5e8-47f4-ba29-0ccb51f4c553 None admin] RBAC: Authorizing `identity:get_project(project_id=86a6b464bc2d4f72895f92c759e26a36)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:14.949485 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0b8b61-a5e8-47f4-ba29-0ccb51f4c553 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:14.951952 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 125/263] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:14 2022] GET /identity/v3/projects/86a6b464bc2d4f72895f92c759e26a36 => generated 303 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:14.953528 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0b8b61-a5e8-47f4-ba29-0ccb51f4c553 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:14.963769 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:14.995159 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:15.093801 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c974843-1c59-4552-8193-2e81519d1a2b None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:15.127590 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0b8b61-a5e8-47f4-ba29-0ccb51f4c553 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:15.138604 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ed6a4c-9d15-4972-9dc2-d3b2eee07a3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:15.139547 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ed6a4c-9d15-4972-9dc2-d3b2eee07a3c None admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:15.140534 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ed6a4c-9d15-4972-9dc2-d3b2eee07a3c None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:15.140821 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ed6a4c-9d15-4972-9dc2-d3b2eee07a3c None admin] PATH_INFO: `/v3/projects/86a6b464bc2d4f72895f92c759e26a36/users/e2cad07e3619484885c0890917feeea4/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:15.142804 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ed6a4c-9d15-4972-9dc2-d3b2eee07a3c None admin] RBAC: Authorizing `identity:create_grant(project_id=86a6b464bc2d4f72895f92c759e26a36, user_id=e2cad07e3619484885c0890917feeea4, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:15.144384 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ed6a4c-9d15-4972-9dc2-d3b2eee07a3c None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:15.159464 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 139/264] 10.222.0.102 () {58 vars in 1444 bytes} [Wed Aug 24 15:33:14 2022] PUT /identity/v3/projects/86a6b464bc2d4f72895f92c759e26a36/users/e2cad07e3619484885c0890917feeea4/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:33:15.160559 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ed6a4c-9d15-4972-9dc2-d3b2eee07a3c None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:15.177297 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15da3111-8fa4-4ac4-81f0-fa830a34c355 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:15.178898 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15da3111-8fa4-4ac4-81f0-fa830a34c355 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:15.179508 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15da3111-8fa4-4ac4-81f0-fa830a34c355 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:15.180132 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15da3111-8fa4-4ac4-81f0-fa830a34c355 None admin] PATH_INFO: `/v3/users/bb05bf01d9d24a5296517f8bde87462b` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:15.182039 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15da3111-8fa4-4ac4-81f0-fa830a34c355 None admin] RBAC: Authorizing `identity:get_user(user_id=bb05bf01d9d24a5296517f8bde87462b)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:15.184797 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15da3111-8fa4-4ac4-81f0-fa830a34c355 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:15.193048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 126/265] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:33:14 2022] GET /identity/v3/users/bb05bf01d9d24a5296517f8bde87462b => generated 284 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:15.193548 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bae7b2d-b82b-4475-bd8b-fbcb8ec7112a None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:15.197435 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bae7b2d-b82b-4475-bd8b-fbcb8ec7112a None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:15.198033 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bae7b2d-b82b-4475-bd8b-fbcb8ec7112a None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:15.200238 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 127/266] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:15 2022] GET /identity => generated 271 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:15.201732 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7ff4a1-3fdc-4269-baeb-fd6b64a9ab80 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:15.202851 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7ff4a1-3fdc-4269-baeb-fd6b64a9ab80 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:15.203257 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7ff4a1-3fdc-4269-baeb-fd6b64a9ab80 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:15.209642 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 128/267] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:15 2022] GET /identity => generated 271 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:15.210943 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7ff4a1-3fdc-4269-baeb-fd6b64a9ab80 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:15.315757 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:15.320274 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:15.332119 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ed6a4c-9d15-4972-9dc2-d3b2eee07a3c None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:15.350709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7ff4a1-3fdc-4269-baeb-fd6b64a9ab80 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:15.378967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8feb9b6f-d5ab-4dd1-8554-d6c53eb26ae7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:15.379605 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8feb9b6f-d5ab-4dd1-8554-d6c53eb26ae7 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:15.379976 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8feb9b6f-d5ab-4dd1-8554-d6c53eb26ae7 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:15.380255 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8feb9b6f-d5ab-4dd1-8554-d6c53eb26ae7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:15.381191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8feb9b6f-d5ab-4dd1-8554-d6c53eb26ae7 None admin] RBAC: Adding query filter params (role.id=4d8fbbaea81a4145b89af12350bb067c, scope.project.id=86a6b464bc2d4f72895f92c759e26a36, user.id=bb05bf01d9d24a5296517f8bde87462b) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:15.381520 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8feb9b6f-d5ab-4dd1-8554-d6c53eb26ae7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:15.383279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8feb9b6f-d5ab-4dd1-8554-d6c53eb26ae7 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:15.393798 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 140/268] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:33:15 2022] GET /identity/v3/role_assignments?user.id=bb05bf01d9d24a5296517f8bde87462b&scope.project.id=86a6b464bc2d4f72895f92c759e26a36&role.id=4d8fbbaea81a4145b89af12350bb067c => generated 610 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:15.395373 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcf5e92-4ec0-43a5-b265-d27cb2dcb676 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:15.399774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcf5e92-4ec0-43a5-b265-d27cb2dcb676 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:15.400915 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261e4df7-9e98-457c-a53d-b2a908be8cb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:15.401512 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcf5e92-4ec0-43a5-b265-d27cb2dcb676 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:15.404441 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261e4df7-9e98-457c-a53d-b2a908be8cb8 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:15.405006 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261e4df7-9e98-457c-a53d-b2a908be8cb8 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:15.406286 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261e4df7-9e98-457c-a53d-b2a908be8cb8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:15.453584 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-cbcf5e92-4ec0-43a5-b265-d27cb2dcb676 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:15.454825 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-cbcf5e92-4ec0-43a5-b265-d27cb2dcb676 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:15.494304 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261e4df7-9e98-457c-a53d-b2a908be8cb8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:15.496366 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261e4df7-9e98-457c-a53d-b2a908be8cb8 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:15.599448 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 141/269] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:15 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:15.600888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d789b95-0b41-4de5-81f4-d6efa0d6ac1a None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:15.601186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d789b95-0b41-4de5-81f4-d6efa0d6ac1a None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:15.601467 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d789b95-0b41-4de5-81f4-d6efa0d6ac1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:15.634106 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:15.641398 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-8d789b95-0b41-4de5-81f4-d6efa0d6ac1a None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:15.642837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-8d789b95-0b41-4de5-81f4-d6efa0d6ac1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:15.669256 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 129/270] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:33:15 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 459 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:33:15.676133 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261e4df7-9e98-457c-a53d-b2a908be8cb8 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:15.809008 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261e4df7-9e98-457c-a53d-b2a908be8cb8 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:15.817501 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 142/271] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:15 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:15.859605 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afc4f66-7f8c-4486-9fdc-056bc39b1fdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:15.860524 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afc4f66-7f8c-4486-9fdc-056bc39b1fdb None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:15.860945 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afc4f66-7f8c-4486-9fdc-056bc39b1fdb None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:15.861361 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afc4f66-7f8c-4486-9fdc-056bc39b1fdb None admin] PATH_INFO: `/v3/projects/86a6b464bc2d4f72895f92c759e26a36` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:15.868148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6afc4f66-7f8c-4486-9fdc-056bc39b1fdb None admin] RBAC: Authorizing `identity:get_project(project_id=86a6b464bc2d4f72895f92c759e26a36)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:15.869887 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6afc4f66-7f8c-4486-9fdc-056bc39b1fdb None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:15.872109 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 130/272] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:15 2022] GET /identity/v3/projects/86a6b464bc2d4f72895f92c759e26a36 => generated 303 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:33:15.878569 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d789b95-0b41-4de5-81f4-d6efa0d6ac1a None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:15.925111 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:16.059507 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d789b95-0b41-4de5-81f4-d6efa0d6ac1a None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:16.117698 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa87f1a-29c7-44e6-a041-12d234d55723 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:16.118321 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa87f1a-29c7-44e6-a041-12d234d55723 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:16.118607 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa87f1a-29c7-44e6-a041-12d234d55723 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:16.118885 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa87f1a-29c7-44e6-a041-12d234d55723 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:16.119649 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa87f1a-29c7-44e6-a041-12d234d55723 None admin] RBAC: Adding query filter params (role.id=b8429ebd9c2a4cbbbbf30ad56ce23bfa, scope.project.id=86a6b464bc2d4f72895f92c759e26a36, user.id=bb05bf01d9d24a5296517f8bde87462b) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:16.119977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa87f1a-29c7-44e6-a041-12d234d55723 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:16.121440 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa87f1a-29c7-44e6-a041-12d234d55723 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:16.132521 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 143/273] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:33:15 2022] GET /identity/v3/role_assignments?user.id=bb05bf01d9d24a5296517f8bde87462b&scope.project.id=86a6b464bc2d4f72895f92c759e26a36&role.id=b8429ebd9c2a4cbbbbf30ad56ce23bfa => generated 610 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:16.150480 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afc4f66-7f8c-4486-9fdc-056bc39b1fdb None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:16.170388 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa87f1a-29c7-44e6-a041-12d234d55723 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:16.236476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:16.298827 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:16.323827 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afc4f66-7f8c-4486-9fdc-056bc39b1fdb None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:16.326412 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa87f1a-29c7-44e6-a041-12d234d55723 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:16.373258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e926dd-e8fb-4298-9a03-3f229da1c9a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:16.374936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e926dd-e8fb-4298-9a03-3f229da1c9a9 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:16.374936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e926dd-e8fb-4298-9a03-3f229da1c9a9 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:16.375149 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e926dd-e8fb-4298-9a03-3f229da1c9a9 None admin] PATH_INFO: `/v3/roles/member` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:16.377979 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c968841-997e-4592-bda6-839e8348c728 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:16.378966 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c968841-997e-4592-bda6-839e8348c728 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:16.379242 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c968841-997e-4592-bda6-839e8348c728 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:16.379511 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c968841-997e-4592-bda6-839e8348c728 None admin] PATH_INFO: `/v3/domains/default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:16.380818 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c968841-997e-4592-bda6-839e8348c728 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:16.382384 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c968841-997e-4592-bda6-839e8348c728 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:16.384055 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 131/274] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:33:16 2022] GET /identity/v3/domains/default => generated 200 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:16.385476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982176cc-42b7-4e95-87fa-652e12a4d9dc None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:16.385763 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982176cc-42b7-4e95-87fa-652e12a4d9dc None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:16.386031 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982176cc-42b7-4e95-87fa-652e12a4d9dc None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:16.387485 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 132/275] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:16 2022] GET /identity => generated 271 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:16.392537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e926dd-e8fb-4298-9a03-3f229da1c9a9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:16.392847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb7237c-5e65-45b9-b5a3-10d19161aedf None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:16.393054 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb7237c-5e65-45b9-b5a3-10d19161aedf None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:16.393420 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb7237c-5e65-45b9-b5a3-10d19161aedf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:16.398318 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-95e926dd-e8fb-4298-9a03-3f229da1c9a9 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:33:16.398318 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:16.398318 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:16.398318 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:16.398318 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:16.398318 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:16.398318 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:16.398318 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:16.398318 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:16.398318 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:16.398318 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:16.398318 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:16.398318 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:16.398318 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:16.398318 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:16.398318 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:16.399340 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:16.399340 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:16.399340 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:16.399340 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:16.399340 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:16.399340 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:16.399340 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:16.399340 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:16.399340 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:16.399340 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:16.399340 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:16.399340 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:16.399340 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:16.399340 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:16.399340 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:16.399340 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:16.400529 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:16.400529 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:16.400529 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:16.400529 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:16.400529 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:16.400529 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:16.400529 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:16.400529 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:16.400529 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:16.400529 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:16.400529 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:33:16.400529 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:16.401114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 144/276] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:33:16 2022] GET /identity/v3/roles/member => generated 84 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:33:16.407273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e926dd-e8fb-4298-9a03-3f229da1c9a9 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:16.423669 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-5cb7237c-5e65-45b9-b5a3-10d19161aedf None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:16.423669 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-5cb7237c-5e65-45b9-b5a3-10d19161aedf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:16.539456 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 133/277] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:16 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 15:33:16.545574 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb7237c-5e65-45b9-b5a3-10d19161aedf None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:16.551234 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e926dd-e8fb-4298-9a03-3f229da1c9a9 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:16.600351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:16.603736 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db3aa68-7057-42b6-9838-02580bc423ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:16.605094 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db3aa68-7057-42b6-9838-02580bc423ed None admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:16.605620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db3aa68-7057-42b6-9838-02580bc423ed None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:16.606606 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db3aa68-7057-42b6-9838-02580bc423ed None admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:16.609036 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db3aa68-7057-42b6-9838-02580bc423ed None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:16.610705 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db3aa68-7057-42b6-9838-02580bc423ed None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:16.630746 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:16.643795 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 145/278] 10.222.0.102 () {60 vars in 1137 bytes} [Wed Aug 24 15:33:16 2022] POST /identity/v3/users => generated 294 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:33:16.691386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb7237c-5e65-45b9-b5a3-10d19161aedf None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:16.735470 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcbf8dd-21c5-44ae-9843-1cebdea39f5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:16.736128 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcbf8dd-21c5-44ae-9843-1cebdea39f5c None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:16.736411 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcbf8dd-21c5-44ae-9843-1cebdea39f5c None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:16.736761 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcbf8dd-21c5-44ae-9843-1cebdea39f5c None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:16.737459 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfcbf8dd-21c5-44ae-9843-1cebdea39f5c None admin] RBAC: Adding query filter params (name=member) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:16.737769 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfcbf8dd-21c5-44ae-9843-1cebdea39f5c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:16.739189 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfcbf8dd-21c5-44ae-9843-1cebdea39f5c None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:16.749779 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 134/279] 10.222.0.102 () {56 vars in 1106 bytes} [Wed Aug 24 15:33:16 2022] GET /identity/v3/roles?name=member => generated 343 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:16.757088 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db3aa68-7057-42b6-9838-02580bc423ed None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:16.821244 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcbf8dd-21c5-44ae-9843-1cebdea39f5c None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:16.897813 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db3aa68-7057-42b6-9838-02580bc423ed None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:16.957540 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9eed400-b1d8-438f-a49d-5cb226820ee8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:16.958359 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9eed400-b1d8-438f-a49d-5cb226820ee8 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:16.958963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9eed400-b1d8-438f-a49d-5cb226820ee8 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:16.959271 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9eed400-b1d8-438f-a49d-5cb226820ee8 None admin] PATH_INFO: `/v3/users/ee02d464635842df93dfc76a01ba29df` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:16.961862 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcbf8dd-21c5-44ae-9843-1cebdea39f5c None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:16.966704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:16.980706 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9eed400-b1d8-438f-a49d-5cb226820ee8 None admin] RBAC: Authorizing `identity:get_user(user_id=ee02d464635842df93dfc76a01ba29df)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:16.982210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9eed400-b1d8-438f-a49d-5cb226820ee8 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:16.985445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 146/280] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:33:16 2022] GET /identity/v3/users/ee02d464635842df93dfc76a01ba29df => generated 294 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:16.992018 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9eed400-b1d8-438f-a49d-5cb226820ee8 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:17.023066 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a721bb-2040-4f91-bb46-6b580f06d2c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:17.024610 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a721bb-2040-4f91-bb46-6b580f06d2c4 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:17.025105 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a721bb-2040-4f91-bb46-6b580f06d2c4 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:17.025381 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a721bb-2040-4f91-bb46-6b580f06d2c4 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:17.037940 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a721bb-2040-4f91-bb46-6b580f06d2c4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:17.043261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-94a721bb-2040-4f91-bb46-6b580f06d2c4 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 15:33:17.043261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:17.043261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:17.043261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:17.043261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:17.043261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:17.043261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:17.043261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:17.043261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:17.043261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:17.043261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:17.043261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:17.043261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:17.043261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:17.043261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:17.043261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:17.044171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:17.044171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:17.044171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:17.044171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:17.044171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:17.044171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:17.044171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:17.044171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:17.044171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:17.044171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:17.044171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:17.044171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:17.044171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:17.044171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:17.044171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:17.044171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:17.044941 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:17.044941 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:17.044941 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:17.044941 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:17.044941 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:17.044941 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:17.044941 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:17.044941 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:17.044941 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:17.044941 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:17.044941 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 15:33:17.044941 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:17.044941 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 135/281] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:33:16 2022] GET /identity/v3/roles/reader => generated 84 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:33:17.051256 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a721bb-2040-4f91-bb46-6b580f06d2c4 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:17.076767 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:17.147531 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9eed400-b1d8-438f-a49d-5cb226820ee8 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:17.195116 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7806cd9-8c4a-45ff-a1fa-604f67f9995f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:17.196125 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7806cd9-8c4a-45ff-a1fa-604f67f9995f None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:17.196843 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7806cd9-8c4a-45ff-a1fa-604f67f9995f None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:17.197548 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7806cd9-8c4a-45ff-a1fa-604f67f9995f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:17.198428 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7806cd9-8c4a-45ff-a1fa-604f67f9995f None admin] RBAC: Adding query filter params (role.id=b5d2ed3e20ab47eeb741b4f43ab2eebb, user.id=ee02d464635842df93dfc76a01ba29df, scope.system=all) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:17.199006 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7806cd9-8c4a-45ff-a1fa-604f67f9995f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:17.201214 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7806cd9-8c4a-45ff-a1fa-604f67f9995f None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:17.203152 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a721bb-2040-4f91-bb46-6b580f06d2c4 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:17.209370 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 147/282] 10.222.0.102 () {56 vars in 1313 bytes} [Wed Aug 24 15:33:16 2022] GET /identity/v3/role_assignments?user.id=ee02d464635842df93dfc76a01ba29df&scope.system=all&role.id=b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 227 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:17.245460 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7ada3a-c141-462e-8ce5-a560a2d0e72a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:17.246074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7ada3a-c141-462e-8ce5-a560a2d0e72a None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:17.246351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7ada3a-c141-462e-8ce5-a560a2d0e72a None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:17.246664 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7ada3a-c141-462e-8ce5-a560a2d0e72a None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:17.247357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7ada3a-c141-462e-8ce5-a560a2d0e72a None admin] RBAC: Adding query filter params (name=reader) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:17.247707 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7ada3a-c141-462e-8ce5-a560a2d0e72a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:17.249688 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7ada3a-c141-462e-8ce5-a560a2d0e72a None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:17.259855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 136/283] 10.222.0.102 () {56 vars in 1106 bytes} [Wed Aug 24 15:33:17 2022] GET /identity/v3/roles?name=reader => generated 343 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:33:17.265803 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7806cd9-8c4a-45ff-a1fa-604f67f9995f None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:17.373671 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718f5ba0-08a5-46f5-9a6e-db61edfcf81e None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:17.374003 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718f5ba0-08a5-46f5-9a6e-db61edfcf81e None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:17.374393 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718f5ba0-08a5-46f5-9a6e-db61edfcf81e None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:17.376684 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 137/284] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:17 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:33:17.399731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34713b5-f56d-4b34-9384-422a6b4f54b0 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:17.399731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34713b5-f56d-4b34-9384-422a6b4f54b0 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:17.399731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34713b5-f56d-4b34-9384-422a6b4f54b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:17.415336 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7806cd9-8c4a-45ff-a1fa-604f67f9995f None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:17.433493 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-a34713b5-f56d-4b34-9384-422a6b4f54b0 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:17.435098 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-a34713b5-f56d-4b34-9384-422a6b4f54b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:17.456594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:17.474979 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:17.479829 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7feeff9-a40b-439a-a198-323aabab142b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:17.480613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7feeff9-a40b-439a-a198-323aabab142b None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:17.481085 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7feeff9-a40b-439a-a198-323aabab142b None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:17.481427 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7feeff9-a40b-439a-a198-323aabab142b None admin] PATH_INFO: `/v3/users/b59f7c2f0d1c42a6816994bf24188271` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:17.482963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7feeff9-a40b-439a-a198-323aabab142b None admin] RBAC: Authorizing `identity:get_user(user_id=b59f7c2f0d1c42a6816994bf24188271)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:17.485749 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7feeff9-a40b-439a-a198-323aabab142b None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:17.488960 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 148/285] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:33:17 2022] GET /identity/v3/users/b59f7c2f0d1c42a6816994bf24188271 => generated 298 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:17.496690 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7feeff9-a40b-439a-a198-323aabab142b None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:17.539780 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 138/286] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:17 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:17.650456 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7feeff9-a40b-439a-a198-323aabab142b None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:17.703448 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb2ca15-f4cd-4c7e-8bdd-1d9f2b9ef476 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:17.704372 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb2ca15-f4cd-4c7e-8bdd-1d9f2b9ef476 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:17.704793 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb2ca15-f4cd-4c7e-8bdd-1d9f2b9ef476 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:17.705233 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb2ca15-f4cd-4c7e-8bdd-1d9f2b9ef476 None admin] PATH_INFO: `/v3/projects/86a6b464bc2d4f72895f92c759e26a36` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:17.712515 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb2ca15-f4cd-4c7e-8bdd-1d9f2b9ef476 None admin] RBAC: Authorizing `identity:get_project(project_id=86a6b464bc2d4f72895f92c759e26a36)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:17.714236 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb2ca15-f4cd-4c7e-8bdd-1d9f2b9ef476 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:17.716482 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 149/287] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:17 2022] GET /identity/v3/projects/86a6b464bc2d4f72895f92c759e26a36 => generated 303 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:17.721923 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34713b5-f56d-4b34-9384-422a6b4f54b0 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:17.832282 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34713b5-f56d-4b34-9384-422a6b4f54b0 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:17.878129 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba9ad8c-da0d-45f8-a496-6db3134cb74b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:17.878467 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb2ca15-f4cd-4c7e-8bdd-1d9f2b9ef476 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:17.878801 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba9ad8c-da0d-45f8-a496-6db3134cb74b None admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:17.879171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba9ad8c-da0d-45f8-a496-6db3134cb74b None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:17.879449 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba9ad8c-da0d-45f8-a496-6db3134cb74b None admin] PATH_INFO: `/v3/projects/86a6b464bc2d4f72895f92c759e26a36/users/b59f7c2f0d1c42a6816994bf24188271/roles/868817cab1414b5aa207bc5bdf4f1d9a` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:17.887395 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba9ad8c-da0d-45f8-a496-6db3134cb74b None admin] RBAC: Authorizing `identity:create_grant(project_id=86a6b464bc2d4f72895f92c759e26a36, user_id=b59f7c2f0d1c42a6816994bf24188271, role_id=868817cab1414b5aa207bc5bdf4f1d9a)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:17.887395 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba9ad8c-da0d-45f8-a496-6db3134cb74b None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:17.909086 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 139/288] 10.222.0.102 () {58 vars in 1444 bytes} [Wed Aug 24 15:33:17 2022] PUT /identity/v3/projects/86a6b464bc2d4f72895f92c759e26a36/users/b59f7c2f0d1c42a6816994bf24188271/roles/868817cab1414b5aa207bc5bdf4f1d9a => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:33:17.962294 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:18.026408 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb2ca15-f4cd-4c7e-8bdd-1d9f2b9ef476 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:18.077839 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6ce085-b82d-47e6-9a44-28f3f094d44c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:18.078501 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6ce085-b82d-47e6-9a44-28f3f094d44c None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:18.078777 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6ce085-b82d-47e6-9a44-28f3f094d44c None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:18.079048 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6ce085-b82d-47e6-9a44-28f3f094d44c None admin] PATH_INFO: `/v3/roles/member` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:18.097069 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6ce085-b82d-47e6-9a44-28f3f094d44c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:18.102001 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-8a6ce085-b82d-47e6-9a44-28f3f094d44c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:33:18.102001 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:18.102001 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:18.102001 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:18.102001 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:18.102001 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:18.102001 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:18.102001 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:18.102001 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:18.102001 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:18.102001 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:18.102001 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:18.102001 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:18.102001 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:18.102001 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:18.102001 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:18.102912 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:18.102912 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:18.102912 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:18.102912 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:18.102912 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:18.102912 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:18.102912 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:18.102912 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:18.102912 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:18.102912 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:18.102912 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:18.102912 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:18.102912 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:18.102912 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:18.102912 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:18.102912 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:18.103717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:18.103717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:18.103717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:18.103717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:18.103717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:18.103717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:18.103717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:18.103717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:18.103717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:18.103717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:18.103717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:33:18.103717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:18.103717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 150/289] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:33:17 2022] GET /identity/v3/roles/member => generated 84 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:18.109958 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba9ad8c-da0d-45f8-a496-6db3134cb74b None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:18.142506 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:18.260728 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba9ad8c-da0d-45f8-a496-6db3134cb74b None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:18.309509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d8d8ca-5c3c-49f6-9e8a-40e7bc230e0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:18.310515 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d8d8ca-5c3c-49f6-9e8a-40e7bc230e0f None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:18.311010 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d8d8ca-5c3c-49f6-9e8a-40e7bc230e0f None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:18.312441 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d8d8ca-5c3c-49f6-9e8a-40e7bc230e0f None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:18.313326 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d8d8ca-5c3c-49f6-9e8a-40e7bc230e0f None admin] RBAC: Adding query filter params (name=member) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:18.313833 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d8d8ca-5c3c-49f6-9e8a-40e7bc230e0f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:18.315413 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d8d8ca-5c3c-49f6-9e8a-40e7bc230e0f None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:18.329620 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 140/290] 10.222.0.102 () {56 vars in 1106 bytes} [Wed Aug 24 15:33:18 2022] GET /identity/v3/roles?name=member => generated 343 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:18.336728 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6ce085-b82d-47e6-9a44-28f3f094d44c None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:18.481897 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6ce085-b82d-47e6-9a44-28f3f094d44c None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:18.500612 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:18.519923 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2932e10f-3ab5-4585-a229-ffad0294fd75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:18.520477 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2932e10f-3ab5-4585-a229-ffad0294fd75 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:18.520804 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2932e10f-3ab5-4585-a229-ffad0294fd75 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:18.521077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2932e10f-3ab5-4585-a229-ffad0294fd75 None admin] PATH_INFO: `/v3/users/e2cad07e3619484885c0890917feeea4` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:18.522215 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2932e10f-3ab5-4585-a229-ffad0294fd75 None admin] RBAC: Authorizing `identity:get_user(user_id=e2cad07e3619484885c0890917feeea4)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:18.523744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2932e10f-3ab5-4585-a229-ffad0294fd75 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:18.525635 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 151/291] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:33:18 2022] GET /identity/v3/users/e2cad07e3619484885c0890917feeea4 => generated 298 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:18.531957 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d8d8ca-5c3c-49f6-9e8a-40e7bc230e0f None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:18.656016 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:18.685695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d8d8ca-5c3c-49f6-9e8a-40e7bc230e0f None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:18.732018 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a33a738-1d60-4091-9501-d6b12ff891a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:18.732636 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a33a738-1d60-4091-9501-d6b12ff891a1 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:18.732888 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a33a738-1d60-4091-9501-d6b12ff891a1 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:18.733275 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a33a738-1d60-4091-9501-d6b12ff891a1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:18.824345 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a33a738-1d60-4091-9501-d6b12ff891a1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:18.824853 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a33a738-1d60-4091-9501-d6b12ff891a1 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:18.940054 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:18.975348 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 141/292] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:33:18 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 444 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:33:18.980919 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2932e10f-3ab5-4585-a229-ffad0294fd75 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:19.153013 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2932e10f-3ab5-4585-a229-ffad0294fd75 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:19.200787 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8687384-ee13-4271-8e62-8cb3bdcaf518 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:19.201428 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8687384-ee13-4271-8e62-8cb3bdcaf518 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:19.201706 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8687384-ee13-4271-8e62-8cb3bdcaf518 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:19.202509 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8687384-ee13-4271-8e62-8cb3bdcaf518 None admin] PATH_INFO: `/v3/projects/86a6b464bc2d4f72895f92c759e26a36` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:19.208337 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8687384-ee13-4271-8e62-8cb3bdcaf518 None admin] RBAC: Authorizing `identity:get_project(project_id=86a6b464bc2d4f72895f92c759e26a36)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:19.209786 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8687384-ee13-4271-8e62-8cb3bdcaf518 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:19.211780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 152/293] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:18 2022] GET /identity/v3/projects/86a6b464bc2d4f72895f92c759e26a36 => generated 303 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:19.218101 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a33a738-1d60-4091-9501-d6b12ff891a1 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:19.356384 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a33a738-1d60-4091-9501-d6b12ff891a1 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:19.407740 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d586458-a5fc-469e-99fb-905c79cb23ab None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:19.407740 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d586458-a5fc-469e-99fb-905c79cb23ab None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:19.407740 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d586458-a5fc-469e-99fb-905c79cb23ab None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:19.408785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 153/294] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:19 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:19.411193 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e3c787-a29f-4443-917c-a605197152d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:19.411842 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e3c787-a29f-4443-917c-a605197152d5 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:19.412126 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e3c787-a29f-4443-917c-a605197152d5 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:19.412402 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e3c787-a29f-4443-917c-a605197152d5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:19.413152 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09e3c787-a29f-4443-917c-a605197152d5 None admin] RBAC: Adding query filter params (role.id=b5d2ed3e20ab47eeb741b4f43ab2eebb, scope.project.id=86a6b464bc2d4f72895f92c759e26a36, user.id=e2cad07e3619484885c0890917feeea4) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:19.413469 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09e3c787-a29f-4443-917c-a605197152d5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:19.414919 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09e3c787-a29f-4443-917c-a605197152d5 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:19.424801 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09973f9e-2cf9-40ab-85ed-ecee196aac2a None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:19.425090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09973f9e-2cf9-40ab-85ed-ecee196aac2a None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:19.425370 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09973f9e-2cf9-40ab-85ed-ecee196aac2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:19.426874 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 142/295] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:33:19 2022] GET /identity/v3/role_assignments?user.id=e2cad07e3619484885c0890917feeea4&scope.project.id=86a6b464bc2d4f72895f92c759e26a36&role.id=b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 610 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:19.452326 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-09973f9e-2cf9-40ab-85ed-ecee196aac2a None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:19.453561 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-09973f9e-2cf9-40ab-85ed-ecee196aac2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:19.532982 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:19.569495 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 154/296] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:19 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 15:33:19.642565 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2f0326-1371-41f6-b104-1932d165697a None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:19.642873 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2f0326-1371-41f6-b104-1932d165697a None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:19.643119 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2f0326-1371-41f6-b104-1932d165697a None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:19.644897 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 143/297] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:19 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:19.650517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b615ff3e-7673-47a5-ac2c-dba55523727f None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:19.650810 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b615ff3e-7673-47a5-ac2c-dba55523727f None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:19.651078 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b615ff3e-7673-47a5-ac2c-dba55523727f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:19.652756 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1478d8b5-de4f-436e-9694-7e444e29d609 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:19.653276 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1478d8b5-de4f-436e-9694-7e444e29d609 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:19.653880 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1478d8b5-de4f-436e-9694-7e444e29d609 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:19.656503 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 155/298] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:19 2022] GET /identity => generated 271 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:19.663125 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1d178f-02bb-4882-96a9-3cc478cf7b54 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:19.663341 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1d178f-02bb-4882-96a9-3cc478cf7b54 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:19.663898 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1d178f-02bb-4882-96a9-3cc478cf7b54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:19.674352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-b615ff3e-7673-47a5-ac2c-dba55523727f None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:19.675817 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-b615ff3e-7673-47a5-ac2c-dba55523727f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:19.694066 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-2d1d178f-02bb-4882-96a9-3cc478cf7b54 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:19.695546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-2d1d178f-02bb-4882-96a9-3cc478cf7b54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:19.717254 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:19.754672 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 144/299] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:19 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 15:33:19.797114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 156/300] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:19 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 15:33:19.945861 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b615ff3e-7673-47a5-ac2c-dba55523727f None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:20.063657 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b615ff3e-7673-47a5-ac2c-dba55523727f None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:20.085050 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1d178f-02bb-4882-96a9-3cc478cf7b54 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:20.104372 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4eaee0-2f4d-4b96-999a-859b2bc1afc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:20.105397 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4eaee0-2f4d-4b96-999a-859b2bc1afc3 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:20.106015 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4eaee0-2f4d-4b96-999a-859b2bc1afc3 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:20.106487 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4eaee0-2f4d-4b96-999a-859b2bc1afc3 None admin] PATH_INFO: `/v3/domains/default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:20.108018 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4eaee0-2f4d-4b96-999a-859b2bc1afc3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:20.111102 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4eaee0-2f4d-4b96-999a-859b2bc1afc3 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:20.113239 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 145/301] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:33:19 2022] GET /identity/v3/domains/default => generated 200 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:20.114340 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4eaee0-2f4d-4b96-999a-859b2bc1afc3 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:20.161847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:20.206406 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1d178f-02bb-4882-96a9-3cc478cf7b54 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:20.221388 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:20.233098 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4eaee0-2f4d-4b96-999a-859b2bc1afc3 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:20.246850 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c39802-e7e5-40d2-a287-8e78c78fa2e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:20.247814 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c39802-e7e5-40d2-a287-8e78c78fa2e4 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:20.248218 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c39802-e7e5-40d2-a287-8e78c78fa2e4 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:20.248669 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c39802-e7e5-40d2-a287-8e78c78fa2e4 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:20.262120 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c39802-e7e5-40d2-a287-8e78c78fa2e4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:20.272168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-00c39802-e7e5-40d2-a287-8e78c78fa2e4 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 15:33:20.272168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:20.272168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:20.272168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:20.272168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:20.272168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:20.272168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:20.272168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:20.272168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:20.272168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:20.272168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:20.272168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:20.272168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:20.272168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:20.272168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:20.272168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:20.273499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:20.273499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:20.273499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:20.273499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:20.273499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:20.273499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:20.273499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:20.273499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:20.273499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:20.273499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:20.273499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:20.273499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:20.273499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:20.273499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:20.273499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:20.273499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:20.275175 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:20.275175 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:20.275175 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:20.275175 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:20.275175 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:20.275175 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:20.275175 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:20.275175 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:20.275175 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:20.275175 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:20.275175 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 15:33:20.275175 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:20.275175 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 157/302] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:33:20 2022] GET /identity/v3/roles/reader => generated 84 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:33:20.275175 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c39802-e7e5-40d2-a287-8e78c78fa2e4 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:20.288154 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a49e50-d611-43da-bafc-8282630829a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:20.289326 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a49e50-d611-43da-bafc-8282630829a8 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:20.289326 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a49e50-d611-43da-bafc-8282630829a8 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:20.289712 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a49e50-d611-43da-bafc-8282630829a8 None admin] PATH_INFO: `/v3/roles/member` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:20.307749 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a49e50-d611-43da-bafc-8282630829a8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:20.316652 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-08a49e50-d611-43da-bafc-8282630829a8 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:33:20.316652 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:20.316652 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:20.316652 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:20.316652 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:20.316652 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:20.316652 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:20.316652 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:20.316652 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:20.316652 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:20.316652 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:20.316652 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:20.316652 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:20.316652 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:20.316652 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:20.316652 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:20.317539 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:20.317539 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:20.317539 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:20.317539 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:20.317539 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:20.317539 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:20.317539 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:20.317539 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:20.317539 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:20.317539 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:20.317539 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:20.317539 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:20.317539 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:20.317539 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:20.317539 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:20.317539 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:20.318321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:20.318321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:20.318321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:20.318321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:20.318321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:20.318321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:20.318321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:20.318321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:20.318321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:20.318321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:20.318321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:33:20.318321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:20.318321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 146/303] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:33:20 2022] GET /identity/v3/roles/member => generated 84 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:20.319932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a49e50-d611-43da-bafc-8282630829a8 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:20.410304 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c39802-e7e5-40d2-a287-8e78c78fa2e4 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:20.441514 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:20.445472 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a49e50-d611-43da-bafc-8282630829a8 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:20.446463 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4405e35-48db-408e-8fa3-fbe781bc0eb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:20.447079 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4405e35-48db-408e-8fa3-fbe781bc0eb0 None admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:20.447356 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4405e35-48db-408e-8fa3-fbe781bc0eb0 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:20.447650 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4405e35-48db-408e-8fa3-fbe781bc0eb0 None admin] PATH_INFO: `/v3/groups` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:20.448285 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4405e35-48db-408e-8fa3-fbe781bc0eb0 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:20.449734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4405e35-48db-408e-8fa3-fbe781bc0eb0 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:20.466212 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 158/304] 10.222.0.102 () {60 vars in 1139 bytes} [Wed Aug 24 15:33:20 2022] POST /identity/v3/groups => generated 230 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:33:20.466964 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4405e35-48db-408e-8fa3-fbe781bc0eb0 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:20.488442 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f45baa64-a7f1-4b31-a870-f6d547e60b10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:20.489063 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45baa64-a7f1-4b31-a870-f6d547e60b10 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:20.489339 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45baa64-a7f1-4b31-a870-f6d547e60b10 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:20.489619 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45baa64-a7f1-4b31-a870-f6d547e60b10 None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:20.490295 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f45baa64-a7f1-4b31-a870-f6d547e60b10 None admin] RBAC: Adding query filter params (name=reader) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:20.490602 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f45baa64-a7f1-4b31-a870-f6d547e60b10 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:20.492042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f45baa64-a7f1-4b31-a870-f6d547e60b10 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:20.507119 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 147/305] 10.222.0.102 () {56 vars in 1106 bytes} [Wed Aug 24 15:33:20 2022] GET /identity/v3/roles?name=reader => generated 343 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:20.507119 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92b1f61-8b5f-4dde-aeaa-b3d911c4c3da None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:20.507119 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92b1f61-8b5f-4dde-aeaa-b3d911c4c3da None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:20.507119 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92b1f61-8b5f-4dde-aeaa-b3d911c4c3da None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:20.507119 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 148/306] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:20 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:20.510850 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92b1f61-8b5f-4dde-aeaa-b3d911c4c3da None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:20.520759 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:20.603890 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4405e35-48db-408e-8fa3-fbe781bc0eb0 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:20.636750 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92b1f61-8b5f-4dde-aeaa-b3d911c4c3da None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:20.644416 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6f6a38-6382-44c3-8894-e99f0e4d1a68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:20.645375 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6f6a38-6382-44c3-8894-e99f0e4d1a68 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:20.645658 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6f6a38-6382-44c3-8894-e99f0e4d1a68 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:20.645932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6f6a38-6382-44c3-8894-e99f0e4d1a68 None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:20.646626 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6f6a38-6382-44c3-8894-e99f0e4d1a68 None admin] RBAC: Adding query filter params (name=member) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:20.646935 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6f6a38-6382-44c3-8894-e99f0e4d1a68 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:20.648596 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6f6a38-6382-44c3-8894-e99f0e4d1a68 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:20.661073 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 159/307] 10.222.0.102 () {56 vars in 1106 bytes} [Wed Aug 24 15:33:20 2022] GET /identity/v3/roles?name=member => generated 343 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:20.662818 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154f6584-7269-4df5-819f-8f296b8f3f1a None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:20.662818 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154f6584-7269-4df5-819f-8f296b8f3f1a None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:20.663100 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154f6584-7269-4df5-819f-8f296b8f3f1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:20.680511 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911f3e9a-eb49-497d-8418-94fb0355a92c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:20.681149 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911f3e9a-eb49-497d-8418-94fb0355a92c None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:20.681579 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911f3e9a-eb49-497d-8418-94fb0355a92c None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:20.681701 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911f3e9a-eb49-497d-8418-94fb0355a92c None admin] PATH_INFO: `/v3/users/8368f63e72a24f0f8e110787042d302e` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:20.682315 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-154f6584-7269-4df5-819f-8f296b8f3f1a None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:20.683538 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-154f6584-7269-4df5-819f-8f296b8f3f1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:20.698100 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-911f3e9a-eb49-497d-8418-94fb0355a92c None admin] RBAC: Authorizing `identity:get_user(user_id=8368f63e72a24f0f8e110787042d302e)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:20.699578 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-911f3e9a-eb49-497d-8418-94fb0355a92c None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:20.702236 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 149/308] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:33:20 2022] GET /identity/v3/users/8368f63e72a24f0f8e110787042d302e => generated 294 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:20.703141 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911f3e9a-eb49-497d-8418-94fb0355a92c None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:20.725841 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:20.763212 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 160/309] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:20 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:20.764853 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154f6584-7269-4df5-819f-8f296b8f3f1a None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:20.809016 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:20.833834 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911f3e9a-eb49-497d-8418-94fb0355a92c None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:20.881243 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500a1a7c-5af2-42e4-ba5d-5318938504e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:20.882236 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500a1a7c-5af2-42e4-ba5d-5318938504e5 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:20.882718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500a1a7c-5af2-42e4-ba5d-5318938504e5 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:20.883310 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500a1a7c-5af2-42e4-ba5d-5318938504e5 None admin] PATH_INFO: `/v3/users/ee02d464635842df93dfc76a01ba29df` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:20.884584 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-500a1a7c-5af2-42e4-ba5d-5318938504e5 None admin] RBAC: Authorizing `identity:get_user(user_id=ee02d464635842df93dfc76a01ba29df)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:20.886233 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-500a1a7c-5af2-42e4-ba5d-5318938504e5 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:20.888065 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 150/310] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:33:20 2022] GET /identity/v3/users/ee02d464635842df93dfc76a01ba29df => generated 294 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:20.895898 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500a1a7c-5af2-42e4-ba5d-5318938504e5 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:20.927446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154f6584-7269-4df5-819f-8f296b8f3f1a None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:20.968497 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad788d1-e103-4b6e-93e9-ee5852a7fe55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:20.969165 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad788d1-e103-4b6e-93e9-ee5852a7fe55 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:20.969452 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad788d1-e103-4b6e-93e9-ee5852a7fe55 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:20.969731 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad788d1-e103-4b6e-93e9-ee5852a7fe55 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:20.970475 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad788d1-e103-4b6e-93e9-ee5852a7fe55 None admin] RBAC: Adding query filter params (role.id=868817cab1414b5aa207bc5bdf4f1d9a, user.id=8368f63e72a24f0f8e110787042d302e, scope.system=all) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:20.970789 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad788d1-e103-4b6e-93e9-ee5852a7fe55 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:20.972499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad788d1-e103-4b6e-93e9-ee5852a7fe55 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:20.986877 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 161/311] 10.222.0.102 () {56 vars in 1313 bytes} [Wed Aug 24 15:33:20 2022] GET /identity/v3/role_assignments?user.id=8368f63e72a24f0f8e110787042d302e&scope.system=all&role.id=868817cab1414b5aa207bc5bdf4f1d9a => generated 227 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:21.011582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:21.032650 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500a1a7c-5af2-42e4-ba5d-5318938504e5 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:21.075353 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31956cb5-12c7-4fcf-94c1-e49c53e99fe1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:21.076039 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31956cb5-12c7-4fcf-94c1-e49c53e99fe1 None admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:21.076322 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31956cb5-12c7-4fcf-94c1-e49c53e99fe1 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:21.076593 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31956cb5-12c7-4fcf-94c1-e49c53e99fe1 None admin] PATH_INFO: `/v3/system/users/ee02d464635842df93dfc76a01ba29df/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:21.078499 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31956cb5-12c7-4fcf-94c1-e49c53e99fe1 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=ee02d464635842df93dfc76a01ba29df, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:21.079975 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31956cb5-12c7-4fcf-94c1-e49c53e99fe1 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:21.091193 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 151/312] 10.222.0.102 () {58 vars in 1339 bytes} [Wed Aug 24 15:33:20 2022] PUT /identity/v3/system/users/ee02d464635842df93dfc76a01ba29df/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:33:21.097799 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad788d1-e103-4b6e-93e9-ee5852a7fe55 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:21.241269 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:21.262041 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad788d1-e103-4b6e-93e9-ee5852a7fe55 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:21.307802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45349349-266c-4727-8e3b-a031be2206d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:21.308470 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45349349-266c-4727-8e3b-a031be2206d3 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:21.308749 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45349349-266c-4727-8e3b-a031be2206d3 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:21.309023 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45349349-266c-4727-8e3b-a031be2206d3 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:21.321020 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45349349-266c-4727-8e3b-a031be2206d3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:21.325776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-45349349-266c-4727-8e3b-a031be2206d3 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 15:33:21.325776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:21.325776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:21.325776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:21.325776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:21.325776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:21.325776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:21.325776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:21.325776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:21.325776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:21.325776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:21.325776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:21.325776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:21.325776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:21.325776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:21.325776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:21.326863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:21.326863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:21.326863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:21.326863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:21.326863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:21.326863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:21.326863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:21.326863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:21.326863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:21.326863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:21.326863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:21.326863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:21.326863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:21.326863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:21.326863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:21.326863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:21.328607 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:21.328607 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:21.328607 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:21.328607 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:21.328607 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:21.328607 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:21.328607 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:21.328607 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:21.328607 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:21.328607 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:21.328607 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 15:33:21.328607 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:21.328607 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 162/313] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:33:21 2022] GET /identity/v3/roles/reader => generated 84 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:21.334672 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31956cb5-12c7-4fcf-94c1-e49c53e99fe1 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:21.451479 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31956cb5-12c7-4fcf-94c1-e49c53e99fe1 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:21.488306 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f6be63-48be-4fad-b6f6-0c674aa2046b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:21.488919 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f6be63-48be-4fad-b6f6-0c674aa2046b None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:21.489203 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f6be63-48be-4fad-b6f6-0c674aa2046b None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:21.489476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f6be63-48be-4fad-b6f6-0c674aa2046b None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:21.490150 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f6be63-48be-4fad-b6f6-0c674aa2046b None admin] RBAC: Adding query filter params (name=reader) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:21.490457 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f6be63-48be-4fad-b6f6-0c674aa2046b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:21.491921 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f6be63-48be-4fad-b6f6-0c674aa2046b None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:21.501691 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 152/314] 10.222.0.102 () {56 vars in 1106 bytes} [Wed Aug 24 15:33:21 2022] GET /identity/v3/roles?name=reader => generated 343 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:21.508827 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45349349-266c-4727-8e3b-a031be2206d3 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:21.622227 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45349349-266c-4727-8e3b-a031be2206d3 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:21.663067 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd3ec12-0887-49b3-a181-c82b78d280d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:21.663741 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd3ec12-0887-49b3-a181-c82b78d280d5 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:21.664027 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd3ec12-0887-49b3-a181-c82b78d280d5 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:21.664302 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd3ec12-0887-49b3-a181-c82b78d280d5 None admin] PATH_INFO: `/v3/users/b59f7c2f0d1c42a6816994bf24188271` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:21.665462 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd3ec12-0887-49b3-a181-c82b78d280d5 None admin] RBAC: Authorizing `identity:get_user(user_id=b59f7c2f0d1c42a6816994bf24188271)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:21.666903 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd3ec12-0887-49b3-a181-c82b78d280d5 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:21.668923 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 163/315] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:33:21 2022] GET /identity/v3/users/b59f7c2f0d1c42a6816994bf24188271 => generated 298 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:21.675323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f6be63-48be-4fad-b6f6-0c674aa2046b None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:21.694379 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:21.791077 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f6be63-48be-4fad-b6f6-0c674aa2046b None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:21.828211 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0006fd3e-f49e-4ea2-86a9-b3e377c53d77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:21.828905 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0006fd3e-f49e-4ea2-86a9-b3e377c53d77 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:21.829189 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0006fd3e-f49e-4ea2-86a9-b3e377c53d77 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:21.829463 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0006fd3e-f49e-4ea2-86a9-b3e377c53d77 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:21.901452 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0006fd3e-f49e-4ea2-86a9-b3e377c53d77 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:21.902936 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0006fd3e-f49e-4ea2-86a9-b3e377c53d77 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:21.922717 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:22.012973 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 153/316] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:33:21 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:33:22.018452 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd3ec12-0887-49b3-a181-c82b78d280d5 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:22.040319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:22.133466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd3ec12-0887-49b3-a181-c82b78d280d5 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:22.173379 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c79593-fb7a-4548-919c-eff43b264c99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:22.174142 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c79593-fb7a-4548-919c-eff43b264c99 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:22.174443 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c79593-fb7a-4548-919c-eff43b264c99 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:22.174741 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c79593-fb7a-4548-919c-eff43b264c99 None admin] PATH_INFO: `/v3/projects/86a6b464bc2d4f72895f92c759e26a36` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:22.176004 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7c79593-fb7a-4548-919c-eff43b264c99 None admin] RBAC: Authorizing `identity:get_project(project_id=86a6b464bc2d4f72895f92c759e26a36)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:22.177468 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7c79593-fb7a-4548-919c-eff43b264c99 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:22.179543 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 164/317] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:22 2022] GET /identity/v3/projects/86a6b464bc2d4f72895f92c759e26a36 => generated 303 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:22.185554 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0006fd3e-f49e-4ea2-86a9-b3e377c53d77 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:22.300030 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0006fd3e-f49e-4ea2-86a9-b3e377c53d77 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:22.315054 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:22.337946 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57bc6291-f40d-4e1f-ac1a-20b439d65250 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:22.338561 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57bc6291-f40d-4e1f-ac1a-20b439d65250 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:22.338842 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57bc6291-f40d-4e1f-ac1a-20b439d65250 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:22.339113 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57bc6291-f40d-4e1f-ac1a-20b439d65250 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:22.339878 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57bc6291-f40d-4e1f-ac1a-20b439d65250 None admin] RBAC: Adding query filter params (role.id=868817cab1414b5aa207bc5bdf4f1d9a, scope.project.id=86a6b464bc2d4f72895f92c759e26a36, user.id=b59f7c2f0d1c42a6816994bf24188271) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:22.340195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57bc6291-f40d-4e1f-ac1a-20b439d65250 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:22.341626 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57bc6291-f40d-4e1f-ac1a-20b439d65250 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:22.348486 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 154/318] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:33:22 2022] GET /identity/v3/role_assignments?user.id=b59f7c2f0d1c42a6816994bf24188271&scope.project.id=86a6b464bc2d4f72895f92c759e26a36&role.id=868817cab1414b5aa207bc5bdf4f1d9a => generated 610 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:22.567807 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455c95da-abdd-4208-9abb-31d139f67d34 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:22.568143 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455c95da-abdd-4208-9abb-31d139f67d34 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:22.568322 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455c95da-abdd-4208-9abb-31d139f67d34 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:22.569644 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 165/319] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:22 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:22.575142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279cee31-2117-4092-950b-23b3d172aba0 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:22.575432 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279cee31-2117-4092-950b-23b3d172aba0 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:22.575775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279cee31-2117-4092-950b-23b3d172aba0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:22.597159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-279cee31-2117-4092-950b-23b3d172aba0 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:22.598366 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-279cee31-2117-4092-950b-23b3d172aba0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:22.672092 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 155/320] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:22 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:22.943360 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455c95da-abdd-4208-9abb-31d139f67d34 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:23.040118 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:23.060026 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455c95da-abdd-4208-9abb-31d139f67d34 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:23.097366 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1891d5bc-02f6-4ebe-93c0-ce66ac6deb16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:23.097925 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1891d5bc-02f6-4ebe-93c0-ce66ac6deb16 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:23.098207 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1891d5bc-02f6-4ebe-93c0-ce66ac6deb16 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:23.098498 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1891d5bc-02f6-4ebe-93c0-ce66ac6deb16 None admin] PATH_INFO: `/v3/domains/default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:23.099589 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1891d5bc-02f6-4ebe-93c0-ce66ac6deb16 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:23.101086 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1891d5bc-02f6-4ebe-93c0-ce66ac6deb16 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:23.102852 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 166/321] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:33:22 2022] GET /identity/v3/domains/default => generated 200 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:33:23.108656 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279cee31-2117-4092-950b-23b3d172aba0 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:23.139019 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409b16b3-e6ee-4c67-887f-51f6cb975fb5 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:23.140228 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409b16b3-e6ee-4c67-887f-51f6cb975fb5 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:23.140228 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409b16b3-e6ee-4c67-887f-51f6cb975fb5 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:23.141518 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 167/322] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:23 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:33:23.142586 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c979abbe-7365-4bad-952b-7d1352fa3e2f None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:23.143305 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c979abbe-7365-4bad-952b-7d1352fa3e2f None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:23.143305 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c979abbe-7365-4bad-952b-7d1352fa3e2f None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:23.144501 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 168/323] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:23 2022] GET /identity => generated 271 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:23.146815 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354b6009-7049-4fdf-9862-af65d5cce941 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:23.147096 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354b6009-7049-4fdf-9862-af65d5cce941 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:23.147373 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354b6009-7049-4fdf-9862-af65d5cce941 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:23.165928 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-354b6009-7049-4fdf-9862-af65d5cce941 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:23.167109 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-354b6009-7049-4fdf-9862-af65d5cce941 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:23.219315 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279cee31-2117-4092-950b-23b3d172aba0 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:23.226822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:23.236026 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 169/324] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:23 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 15:33:23.241387 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c294070-43ca-4058-b7ae-2046a918a49a None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:23.241676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c294070-43ca-4058-b7ae-2046a918a49a None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:23.242105 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c294070-43ca-4058-b7ae-2046a918a49a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:23.257070 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19efac40-f139-4e8a-ae26-dd72aec316e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:23.257759 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19efac40-f139-4e8a-ae26-dd72aec316e5 None admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:23.257974 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19efac40-f139-4e8a-ae26-dd72aec316e5 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:23.258249 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19efac40-f139-4e8a-ae26-dd72aec316e5 None admin] PATH_INFO: `/v3/groups` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:23.258898 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19efac40-f139-4e8a-ae26-dd72aec316e5 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:23.261962 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-8c294070-43ca-4058-b7ae-2046a918a49a None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:23.262317 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19efac40-f139-4e8a-ae26-dd72aec316e5 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:23.262840 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-8c294070-43ca-4058-b7ae-2046a918a49a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:23.284295 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 156/325] 10.222.0.102 () {60 vars in 1139 bytes} [Wed Aug 24 15:33:23 2022] POST /identity/v3/groups => generated 227 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:33:23.344502 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 170/326] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:23 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:23.514224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19efac40-f139-4e8a-ae26-dd72aec316e5 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:23.615200 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c294070-43ca-4058-b7ae-2046a918a49a None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:23.632888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:23.641180 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19efac40-f139-4e8a-ae26-dd72aec316e5 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:23.683319 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3db233b-2fe5-43f0-b50a-b1267c8d1eb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:23.683518 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3db233b-2fe5-43f0-b50a-b1267c8d1eb4 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:23.683828 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3db233b-2fe5-43f0-b50a-b1267c8d1eb4 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:23.684102 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3db233b-2fe5-43f0-b50a-b1267c8d1eb4 None admin] PATH_INFO: `/v3/roles/member` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:23.703476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3db233b-2fe5-43f0-b50a-b1267c8d1eb4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:23.709310 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-e3db233b-2fe5-43f0-b50a-b1267c8d1eb4 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:33:23.709310 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:23.709310 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:23.709310 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:23.709310 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:23.709310 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:23.709310 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:23.709310 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:23.709310 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:23.709310 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:23.709310 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:23.709310 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:23.709310 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:23.709310 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:23.709310 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:23.709310 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:23.710196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:23.710196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:23.710196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:23.710196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:23.710196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:23.710196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:23.710196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:23.710196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:23.710196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:23.710196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:23.710196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:23.710196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:23.710196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:23.710196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:23.710196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:23.710196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:23.710977 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:23.710977 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:23.710977 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:23.710977 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:23.710977 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:23.710977 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:23.710977 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:23.710977 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:23.710977 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:23.710977 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:23.710977 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:33:23.710977 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:23.710977 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 157/327] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:33:23 2022] GET /identity/v3/roles/member => generated 84 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:23.716736 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3db233b-2fe5-43f0-b50a-b1267c8d1eb4 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:23.756134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:23.768217 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c294070-43ca-4058-b7ae-2046a918a49a None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:23.815168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a094b7-8281-4a04-9d3b-a44939c3bc2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:23.816174 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a094b7-8281-4a04-9d3b-a44939c3bc2e None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:23.816725 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a094b7-8281-4a04-9d3b-a44939c3bc2e None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:23.817158 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a094b7-8281-4a04-9d3b-a44939c3bc2e None admin] PATH_INFO: `/v3/roles/reader` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:23.834159 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a094b7-8281-4a04-9d3b-a44939c3bc2e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:23.840341 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-64a094b7-8281-4a04-9d3b-a44939c3bc2e None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 15:33:23.840341 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:23.840341 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:23.840341 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:23.840341 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:23.840341 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:23.840341 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:23.840341 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:23.840341 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:23.840341 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:23.840341 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:23.840341 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:23.840341 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:23.840341 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:23.840341 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:23.840341 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:23.841446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:23.841446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:23.841446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:23.841446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:23.841446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:23.841446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:23.841446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:23.841446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:23.841446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:23.841446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:23.841446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:23.841446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:23.841446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:23.841446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:23.841446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:23.841446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:23.843168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:23.843168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:23.843168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:23.843168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:23.843168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:23.843168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:23.843168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:23.843168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:23.843168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:23.843168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:23.843168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 15:33:23.843168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:23.846203 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 171/328] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:33:23 2022] GET /identity/v3/roles/reader => generated 84 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:33:23.852912 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a094b7-8281-4a04-9d3b-a44939c3bc2e None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:23.862184 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3db233b-2fe5-43f0-b50a-b1267c8d1eb4 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:23.915451 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efdf9b8f-b2a0-424b-8df7-87220d19ea6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:23.916662 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdf9b8f-b2a0-424b-8df7-87220d19ea6d None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:23.917200 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdf9b8f-b2a0-424b-8df7-87220d19ea6d None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:23.918098 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdf9b8f-b2a0-424b-8df7-87220d19ea6d None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:23.919021 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efdf9b8f-b2a0-424b-8df7-87220d19ea6d None admin] RBAC: Adding query filter params (name=member) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:23.919534 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efdf9b8f-b2a0-424b-8df7-87220d19ea6d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:23.923162 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efdf9b8f-b2a0-424b-8df7-87220d19ea6d None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:23.937914 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 158/329] 10.222.0.102 () {56 vars in 1106 bytes} [Wed Aug 24 15:33:23 2022] GET /identity/v3/roles?name=member => generated 343 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:23.943825 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efdf9b8f-b2a0-424b-8df7-87220d19ea6d None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:23.944482 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:23.990332 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a094b7-8281-4a04-9d3b-a44939c3bc2e None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:24.032764 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af699541-e8cc-4d95-87a9-28a6fa8f911e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:24.033300 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af699541-e8cc-4d95-87a9-28a6fa8f911e None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:24.033586 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af699541-e8cc-4d95-87a9-28a6fa8f911e None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:24.034125 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af699541-e8cc-4d95-87a9-28a6fa8f911e None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:24.034529 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af699541-e8cc-4d95-87a9-28a6fa8f911e None admin] RBAC: Adding query filter params (name=reader) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:24.034832 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af699541-e8cc-4d95-87a9-28a6fa8f911e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:24.036282 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af699541-e8cc-4d95-87a9-28a6fa8f911e None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:24.046060 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 172/330] 10.222.0.102 () {56 vars in 1106 bytes} [Wed Aug 24 15:33:23 2022] GET /identity/v3/roles?name=reader => generated 343 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:33:24.057932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af699541-e8cc-4d95-87a9-28a6fa8f911e None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:24.075209 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:24.090351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efdf9b8f-b2a0-424b-8df7-87220d19ea6d None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:24.137838 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc076580-c1a1-43e4-bfd9-94fc4736f244 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:24.138633 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc076580-c1a1-43e4-bfd9-94fc4736f244 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:24.138819 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc076580-c1a1-43e4-bfd9-94fc4736f244 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:24.139115 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc076580-c1a1-43e4-bfd9-94fc4736f244 None admin] PATH_INFO: `/v3/users/ee02d464635842df93dfc76a01ba29df` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:24.141722 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc076580-c1a1-43e4-bfd9-94fc4736f244 None admin] RBAC: Authorizing `identity:get_user(user_id=ee02d464635842df93dfc76a01ba29df)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:24.143324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc076580-c1a1-43e4-bfd9-94fc4736f244 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:24.145338 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 159/331] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:33:23 2022] GET /identity/v3/users/ee02d464635842df93dfc76a01ba29df => generated 294 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:33:24.152922 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc076580-c1a1-43e4-bfd9-94fc4736f244 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:24.198195 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af699541-e8cc-4d95-87a9-28a6fa8f911e None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:24.248431 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95dd3a9c-da0a-4c62-b6c5-04c460d602fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:24.249715 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95dd3a9c-da0a-4c62-b6c5-04c460d602fb None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:24.250200 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95dd3a9c-da0a-4c62-b6c5-04c460d602fb None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:24.250699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95dd3a9c-da0a-4c62-b6c5-04c460d602fb None admin] PATH_INFO: `/v3/users/8368f63e72a24f0f8e110787042d302e` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:24.252422 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95dd3a9c-da0a-4c62-b6c5-04c460d602fb None admin] RBAC: Authorizing `identity:get_user(user_id=8368f63e72a24f0f8e110787042d302e)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:24.254739 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95dd3a9c-da0a-4c62-b6c5-04c460d602fb None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:24.257470 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 173/332] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:33:24 2022] GET /identity/v3/users/8368f63e72a24f0f8e110787042d302e => generated 294 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:33:24.263212 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95dd3a9c-da0a-4c62-b6c5-04c460d602fb None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:24.268608 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:24.294170 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc076580-c1a1-43e4-bfd9-94fc4736f244 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:24.334654 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae738e2-800f-482e-8852-fc943136eba2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:24.335262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae738e2-800f-482e-8852-fc943136eba2 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:24.335542 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae738e2-800f-482e-8852-fc943136eba2 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:24.335868 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae738e2-800f-482e-8852-fc943136eba2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:24.336603 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae738e2-800f-482e-8852-fc943136eba2 None admin] RBAC: Adding query filter params (role.id=b5d2ed3e20ab47eeb741b4f43ab2eebb, user.id=ee02d464635842df93dfc76a01ba29df, scope.system=all) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:24.336910 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae738e2-800f-482e-8852-fc943136eba2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:24.338342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae738e2-800f-482e-8852-fc943136eba2 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:24.353485 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 160/333] 10.222.0.102 () {56 vars in 1313 bytes} [Wed Aug 24 15:33:24 2022] GET /identity/v3/role_assignments?user.id=ee02d464635842df93dfc76a01ba29df&scope.system=all&role.id=b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 512 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:24.384020 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95dd3a9c-da0a-4c62-b6c5-04c460d602fb None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:24.425263 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903c5eba-433f-461e-8ce1-38aa2df8f698 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:24.426334 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903c5eba-433f-461e-8ce1-38aa2df8f698 None admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:24.426817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903c5eba-433f-461e-8ce1-38aa2df8f698 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:24.427314 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903c5eba-433f-461e-8ce1-38aa2df8f698 None admin] PATH_INFO: `/v3/system/users/8368f63e72a24f0f8e110787042d302e/roles/868817cab1414b5aa207bc5bdf4f1d9a` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:24.429182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903c5eba-433f-461e-8ce1-38aa2df8f698 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=8368f63e72a24f0f8e110787042d302e, role_id=868817cab1414b5aa207bc5bdf4f1d9a)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:24.430845 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903c5eba-433f-461e-8ce1-38aa2df8f698 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:24.442082 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 174/334] 10.222.0.102 () {58 vars in 1339 bytes} [Wed Aug 24 15:33:24 2022] PUT /identity/v3/system/users/8368f63e72a24f0f8e110787042d302e/roles/868817cab1414b5aa207bc5bdf4f1d9a => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 15:33:25.772993 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd61144-1193-4e6a-a811-5ff0dd012bec None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:25.773619 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd61144-1193-4e6a-a811-5ff0dd012bec None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:25.773619 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd61144-1193-4e6a-a811-5ff0dd012bec None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:25.775108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 161/335] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:25 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:25.782263 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ace37b-dc94-4fb7-bbbe-6f8a45e42c3b None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:25.783731 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ace37b-dc94-4fb7-bbbe-6f8a45e42c3b None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:25.784220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ace37b-dc94-4fb7-bbbe-6f8a45e42c3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:25.813414 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-d1ace37b-dc94-4fb7-bbbe-6f8a45e42c3b None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:25.814965 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-d1ace37b-dc94-4fb7-bbbe-6f8a45e42c3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:25.912968 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:25.925065 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 175/336] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:25 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:25.950735 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d8898e-911e-4759-83b9-c5bf0e8ec698 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:25.951071 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d8898e-911e-4759-83b9-c5bf0e8ec698 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:25.951289 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d8898e-911e-4759-83b9-c5bf0e8ec698 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:25.953458 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 162/337] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:25 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:25.959308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33249ed1-f731-41ee-bfcc-dbc99bffb87b None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:25.959600 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33249ed1-f731-41ee-bfcc-dbc99bffb87b None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:25.959977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33249ed1-f731-41ee-bfcc-dbc99bffb87b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:25.991504 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-33249ed1-f731-41ee-bfcc-dbc99bffb87b None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:25.993303 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-33249ed1-f731-41ee-bfcc-dbc99bffb87b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:26.089462 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 176/338] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:25 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:26.362168 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d8898e-911e-4759-83b9-c5bf0e8ec698 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:26.381641 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:26.405441 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33249ed1-f731-41ee-bfcc-dbc99bffb87b None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:26.557718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d8898e-911e-4759-83b9-c5bf0e8ec698 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:26.582709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33249ed1-f731-41ee-bfcc-dbc99bffb87b None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:26.635742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc63e86-8f84-4590-824c-e5ee469243e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:26.637008 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc63e86-8f84-4590-824c-e5ee469243e6 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:26.638309 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc63e86-8f84-4590-824c-e5ee469243e6 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:26.639983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc63e86-8f84-4590-824c-e5ee469243e6 None admin] PATH_INFO: `/v3/roles/member` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:26.643868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:26.664746 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63cb906-1371-4ed1-9720-1dc0f81e3b57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:26.665908 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63cb906-1371-4ed1-9720-1dc0f81e3b57 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:26.666402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63cb906-1371-4ed1-9720-1dc0f81e3b57 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:26.666928 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63cb906-1371-4ed1-9720-1dc0f81e3b57 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:26.668670 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc63e86-8f84-4590-824c-e5ee469243e6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:26.674726 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-5fc63e86-8f84-4590-824c-e5ee469243e6 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:33:26.674726 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:26.674726 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:26.674726 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:26.674726 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:26.674726 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:26.674726 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:26.674726 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:26.674726 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:26.674726 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:26.674726 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:26.674726 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:26.674726 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:26.674726 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:26.674726 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:26.674726 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:26.675820 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:26.675820 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:26.675820 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:26.675820 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:26.675820 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:26.675820 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:26.675820 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:26.675820 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:26.675820 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:26.675820 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:26.675820 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:26.675820 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:26.675820 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:26.675820 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:26.675820 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:26.675820 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:26.676637 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:26.676637 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:26.676637 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:26.676637 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:26.676637 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:26.676637 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:26.676637 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:26.676637 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:26.676637 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:26.676637 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:26.676637 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:33:26.676637 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:26.685333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 163/339] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:33:26 2022] GET /identity/v3/roles/member => generated 84 bytes in 323 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:26.692518 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc63e86-8f84-4590-824c-e5ee469243e6 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:26.694529 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f63cb906-1371-4ed1-9720-1dc0f81e3b57 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:26.706400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-f63cb906-1371-4ed1-9720-1dc0f81e3b57 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 15:33:26.706400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:26.706400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:26.706400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:26.706400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:26.706400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:26.706400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:26.706400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:26.706400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:26.706400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:26.706400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:26.706400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:26.706400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:26.706400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:26.706400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:26.706400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:26.707466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:26.707466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:26.707466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:26.707466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:26.707466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:26.707466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:26.707466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:26.707466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:26.707466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:26.707466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:26.707466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:26.707466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:26.707466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:26.707466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:26.707466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:26.707466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:26.708384 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:26.708384 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:26.708384 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:26.708384 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:26.708384 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:26.708384 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:26.708384 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:26.708384 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:26.708384 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:26.708384 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:26.708384 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 15:33:26.708384 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:26.720020 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 177/340] 10.222.0.102 () {56 vars in 1119 bytes} [Wed Aug 24 15:33:26 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 315 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:33:26.741979 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63cb906-1371-4ed1-9720-1dc0f81e3b57 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:26.801718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:26.834378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc63e86-8f84-4590-824c-e5ee469243e6 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:26.893343 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e963c6-4e34-4674-a6f7-92773e6bc29f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:26.894588 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e963c6-4e34-4674-a6f7-92773e6bc29f None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:26.895081 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e963c6-4e34-4674-a6f7-92773e6bc29f None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:26.895556 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e963c6-4e34-4674-a6f7-92773e6bc29f None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:26.896497 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37e963c6-4e34-4674-a6f7-92773e6bc29f None admin] RBAC: Adding query filter params (name=member) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:26.897313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37e963c6-4e34-4674-a6f7-92773e6bc29f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:26.903769 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37e963c6-4e34-4674-a6f7-92773e6bc29f None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:26.921429 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 164/341] 10.222.0.102 () {56 vars in 1106 bytes} [Wed Aug 24 15:33:26 2022] GET /identity/v3/roles?name=member => generated 343 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:26.927164 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e963c6-4e34-4674-a6f7-92773e6bc29f None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:26.977879 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63cb906-1371-4ed1-9720-1dc0f81e3b57 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:27.067293 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0bd1fc-d18d-4ab7-bf1d-ca9870b1c3ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:27.067952 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0bd1fc-d18d-4ab7-bf1d-ca9870b1c3ea None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:27.068231 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0bd1fc-d18d-4ab7-bf1d-ca9870b1c3ea None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:27.068496 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0bd1fc-d18d-4ab7-bf1d-ca9870b1c3ea None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:27.069169 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f0bd1fc-d18d-4ab7-bf1d-ca9870b1c3ea None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:27.069469 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f0bd1fc-d18d-4ab7-bf1d-ca9870b1c3ea None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:27.070854 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f0bd1fc-d18d-4ab7-bf1d-ca9870b1c3ea None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:27.073132 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e963c6-4e34-4674-a6f7-92773e6bc29f None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:27.087097 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 178/342] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:33:26 2022] GET /identity/v3/roles?name=anotherrole => generated 336 bytes in 345 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:27.101459 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0bd1fc-d18d-4ab7-bf1d-ca9870b1c3ea None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:27.124601 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aecf5c8-6bfd-4291-9a17-3f5aac2c8239 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:27.125230 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aecf5c8-6bfd-4291-9a17-3f5aac2c8239 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:27.125511 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aecf5c8-6bfd-4291-9a17-3f5aac2c8239 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:27.125783 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aecf5c8-6bfd-4291-9a17-3f5aac2c8239 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:27.133938 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:27.221648 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aecf5c8-6bfd-4291-9a17-3f5aac2c8239 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:27.223056 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aecf5c8-6bfd-4291-9a17-3f5aac2c8239 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:27.225484 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:27.338694 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 165/343] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:33:26 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 412 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:33:27.340034 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3e0b8f-391b-4c76-a1c8-f66f853da7b1 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:27.340324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3e0b8f-391b-4c76-a1c8-f66f853da7b1 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:27.340602 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3e0b8f-391b-4c76-a1c8-f66f853da7b1 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:27.342177 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 166/344] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:27 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:27.349419 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3e0b8f-391b-4c76-a1c8-f66f853da7b1 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:27.358797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0bd1fc-d18d-4ab7-bf1d-ca9870b1c3ea None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:27.395481 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:27.412321 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdb6507-e25d-4634-9671-f9c063c39426 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:27.413318 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdb6507-e25d-4634-9671-f9c063c39426 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:27.413806 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdb6507-e25d-4634-9671-f9c063c39426 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:27.414290 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdb6507-e25d-4634-9671-f9c063c39426 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:27.493838 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3e0b8f-391b-4c76-a1c8-f66f853da7b1 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:27.516291 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdb6507-e25d-4634-9671-f9c063c39426 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:27.517769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdb6507-e25d-4634-9671-f9c063c39426 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:27.545041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aaec36f-5693-4cf2-9950-6ebd2f8586e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:27.546578 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aaec36f-5693-4cf2-9950-6ebd2f8586e7 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:27.547309 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aaec36f-5693-4cf2-9950-6ebd2f8586e7 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:27.547886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aaec36f-5693-4cf2-9950-6ebd2f8586e7 None admin] PATH_INFO: `/v3/projects/749c6d651fc445cdb75db25a73873386` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:27.550065 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aaec36f-5693-4cf2-9950-6ebd2f8586e7 None admin] RBAC: Authorizing `identity:get_project(project_id=749c6d651fc445cdb75db25a73873386)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:27.552396 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aaec36f-5693-4cf2-9950-6ebd2f8586e7 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:27.555111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 167/345] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:27 2022] GET /identity/v3/projects/749c6d651fc445cdb75db25a73873386 => generated 299 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:27.557512 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c090160a-e797-4fe0-b996-6608e9a3ed82 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:27.558018 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c090160a-e797-4fe0-b996-6608e9a3ed82 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:27.558527 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c090160a-e797-4fe0-b996-6608e9a3ed82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:27.578064 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:27.584000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-c090160a-e797-4fe0-b996-6608e9a3ed82 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:27.585247 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-c090160a-e797-4fe0-b996-6608e9a3ed82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:27.653416 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 179/346] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:33:27 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 552 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 24 15:33:27.656430 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0968f5b5-325c-4c21-bb04-466b60992bec None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:27.657247 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0968f5b5-325c-4c21-bb04-466b60992bec None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:27.657742 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0968f5b5-325c-4c21-bb04-466b60992bec None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:27.660047 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 180/347] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:27 2022] GET /identity => generated 271 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:27.661123 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0968f5b5-325c-4c21-bb04-466b60992bec None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:27.675353 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 168/348] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:27 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:27.677011 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c090160a-e797-4fe0-b996-6608e9a3ed82 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:27.725540 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:27.835750 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0968f5b5-325c-4c21-bb04-466b60992bec None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:27.883319 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c090160a-e797-4fe0-b996-6608e9a3ed82 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:27.904443 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0a7f2f-2d76-404f-ac24-0681826fb480 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:27.905791 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0a7f2f-2d76-404f-ac24-0681826fb480 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:27.907060 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0a7f2f-2d76-404f-ac24-0681826fb480 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:27.907463 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0a7f2f-2d76-404f-ac24-0681826fb480 None admin] PATH_INFO: `/v3/groups/6d47b38427704c8baaaefd6a3ae615a6` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:27.918234 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a0a7f2f-2d76-404f-ac24-0681826fb480 None admin] RBAC: Authorizing `identity:get_group(group_id=6d47b38427704c8baaaefd6a3ae615a6)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:27.922340 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a0a7f2f-2d76-404f-ac24-0681826fb480 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:27.925359 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 181/349] 10.222.0.102 () {56 vars in 1185 bytes} [Wed Aug 24 15:33:27 2022] GET /identity/v3/groups/6d47b38427704c8baaaefd6a3ae615a6 => generated 227 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:27.937772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f5fc32-c27c-4d85-9b96-7e6ada7b06c9 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:27.938209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f5fc32-c27c-4d85-9b96-7e6ada7b06c9 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:27.938594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f5fc32-c27c-4d85-9b96-7e6ada7b06c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:27.970898 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-f6f5fc32-c27c-4d85-9b96-7e6ada7b06c9 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:27.979900 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44fd9480-ea17-4757-8ff5-cc51f9b46b55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:27.979900 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fd9480-ea17-4757-8ff5-cc51f9b46b55 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:27.979900 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fd9480-ea17-4757-8ff5-cc51f9b46b55 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:27.979900 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fd9480-ea17-4757-8ff5-cc51f9b46b55 None admin] PATH_INFO: `/v3/projects/749c6d651fc445cdb75db25a73873386` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:27.980322 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44fd9480-ea17-4757-8ff5-cc51f9b46b55 None admin] RBAC: Authorizing `identity:get_project(project_id=749c6d651fc445cdb75db25a73873386)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:27.980322 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44fd9480-ea17-4757-8ff5-cc51f9b46b55 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:27.981860 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-f6f5fc32-c27c-4d85-9b96-7e6ada7b06c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:27.986402 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 169/350] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:27 2022] GET /identity/v3/projects/749c6d651fc445cdb75db25a73873386 => generated 299 bytes in 310 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:27.987334 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44fd9480-ea17-4757-8ff5-cc51f9b46b55 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:27.998064 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:28.145610 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 182/351] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:27 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:28.146983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f5fc32-c27c-4d85-9b96-7e6ada7b06c9 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:28.187653 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44fd9480-ea17-4757-8ff5-cc51f9b46b55 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:28.188118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:28.234796 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfdc889-2cb2-4819-af05-6f43906a8610 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:28.235508 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfdc889-2cb2-4819-af05-6f43906a8610 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:28.239653 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfdc889-2cb2-4819-af05-6f43906a8610 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:28.239653 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfdc889-2cb2-4819-af05-6f43906a8610 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:28.239653 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dfdc889-2cb2-4819-af05-6f43906a8610 None admin] RBAC: Adding query filter params (group.id=6d47b38427704c8baaaefd6a3ae615a6, role.id=b5d2ed3e20ab47eeb741b4f43ab2eebb, scope.project.id=749c6d651fc445cdb75db25a73873386) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:28.239653 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dfdc889-2cb2-4819-af05-6f43906a8610 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:28.239653 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dfdc889-2cb2-4819-af05-6f43906a8610 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:28.245858 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 170/352] 10.222.0.102 () {56 vars in 1381 bytes} [Wed Aug 24 15:33:27 2022] GET /identity/v3/role_assignments?group.id=6d47b38427704c8baaaefd6a3ae615a6&scope.project.id=749c6d651fc445cdb75db25a73873386&role.id=b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 261 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:28.246892 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfdc889-2cb2-4819-af05-6f43906a8610 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:28.311935 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f5fc32-c27c-4d85-9b96-7e6ada7b06c9 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:28.383514 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b85e5a8-f585-419b-a3ca-5fde55a0b311 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:28.384205 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b85e5a8-f585-419b-a3ca-5fde55a0b311 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:28.384492 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b85e5a8-f585-419b-a3ca-5fde55a0b311 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:28.384769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b85e5a8-f585-419b-a3ca-5fde55a0b311 None admin] PATH_INFO: `/v3/groups/6d47b38427704c8baaaefd6a3ae615a6` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:28.385920 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b85e5a8-f585-419b-a3ca-5fde55a0b311 None admin] RBAC: Authorizing `identity:get_group(group_id=6d47b38427704c8baaaefd6a3ae615a6)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:28.387396 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b85e5a8-f585-419b-a3ca-5fde55a0b311 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:28.396371 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 183/353] 10.222.0.102 () {56 vars in 1185 bytes} [Wed Aug 24 15:33:28 2022] GET /identity/v3/groups/6d47b38427704c8baaaefd6a3ae615a6 => generated 227 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:28.397789 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee04547-7b8e-423b-9266-eecd1cb68c72 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:28.398039 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee04547-7b8e-423b-9266-eecd1cb68c72 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:28.399673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee04547-7b8e-423b-9266-eecd1cb68c72 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:28.399934 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 184/354] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:28 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:28.409959 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63da19ed-0fb4-423f-80a5-43296b8dcb4b None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:28.410281 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63da19ed-0fb4-423f-80a5-43296b8dcb4b None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:28.410763 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63da19ed-0fb4-423f-80a5-43296b8dcb4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:28.427663 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfdc889-2cb2-4819-af05-6f43906a8610 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:28.463011 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-63da19ed-0fb4-423f-80a5-43296b8dcb4b None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:28.464369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-63da19ed-0fb4-423f-80a5-43296b8dcb4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:28.475332 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:28.550964 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746777b2-1a5d-46d8-99d3-08e0c941e1df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:28.551621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746777b2-1a5d-46d8-99d3-08e0c941e1df None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:28.551933 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746777b2-1a5d-46d8-99d3-08e0c941e1df None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:28.552201 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746777b2-1a5d-46d8-99d3-08e0c941e1df None admin] PATH_INFO: `/v3/roles/member` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:28.571536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746777b2-1a5d-46d8-99d3-08e0c941e1df None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:28.578676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-746777b2-1a5d-46d8-99d3-08e0c941e1df None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:33:28.578676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:28.578676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:28.578676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:28.578676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:28.578676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:28.578676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:28.578676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:28.578676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:28.578676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:28.578676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:28.578676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:28.578676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:28.578676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:28.578676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:28.578676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:28.579875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:28.579875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:28.579875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:28.579875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:28.579875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:28.579875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:28.579875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:28.579875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:28.579875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:28.579875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:28.579875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:28.579875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:28.579875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:28.579875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:28.579875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:28.579875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:28.580822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:28.580822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:28.580822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:28.580822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:28.580822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:28.580822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:28.580822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:28.580822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:28.580822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:28.580822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:28.580822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:33:28.580822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:28.581425 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 171/355] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:33:28 2022] GET /identity/v3/roles/member => generated 84 bytes in 334 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:28.581941 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746777b2-1a5d-46d8-99d3-08e0c941e1df None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:28.615803 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:28.620709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 185/356] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:28 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 15:33:28.621536 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63da19ed-0fb4-423f-80a5-43296b8dcb4b None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:28.735202 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746777b2-1a5d-46d8-99d3-08e0c941e1df None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:28.761911 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63da19ed-0fb4-423f-80a5-43296b8dcb4b None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:28.776339 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7902198-6ca6-4489-9d95-515a1ad0069c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:28.777195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7902198-6ca6-4489-9d95-515a1ad0069c None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:28.777524 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7902198-6ca6-4489-9d95-515a1ad0069c None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:28.778175 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7902198-6ca6-4489-9d95-515a1ad0069c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:28.778902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7902198-6ca6-4489-9d95-515a1ad0069c None admin] RBAC: Adding query filter params (group.id=6d47b38427704c8baaaefd6a3ae615a6, role.id=b8429ebd9c2a4cbbbbf30ad56ce23bfa, scope.project.id=749c6d651fc445cdb75db25a73873386) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:28.779292 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7902198-6ca6-4489-9d95-515a1ad0069c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:28.781770 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7902198-6ca6-4489-9d95-515a1ad0069c None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:28.790098 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 172/357] 10.222.0.102 () {56 vars in 1381 bytes} [Wed Aug 24 15:33:28 2022] GET /identity/v3/role_assignments?group.id=6d47b38427704c8baaaefd6a3ae615a6&scope.project.id=749c6d651fc445cdb75db25a73873386&role.id=b8429ebd9c2a4cbbbbf30ad56ce23bfa => generated 261 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:28.791800 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434ed11c-abfa-4d73-8e8b-5e5b25a85305 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:28.791800 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434ed11c-abfa-4d73-8e8b-5e5b25a85305 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:28.792059 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434ed11c-abfa-4d73-8e8b-5e5b25a85305 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:28.793477 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 173/358] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:28 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:28.802253 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434ed11c-abfa-4d73-8e8b-5e5b25a85305 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:28.820171 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14fc9b1d-6c16-43a5-a764-d960348f8e37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:28.827925 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fc9b1d-6c16-43a5-a764-d960348f8e37 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:28.828112 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fc9b1d-6c16-43a5-a764-d960348f8e37 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:28.828315 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fc9b1d-6c16-43a5-a764-d960348f8e37 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:28.857211 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:28.866032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14fc9b1d-6c16-43a5-a764-d960348f8e37 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:28.872241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-14fc9b1d-6c16-43a5-a764-d960348f8e37 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 15:33:28.872241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:28.872241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:28.872241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:28.872241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:28.872241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:28.872241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:28.872241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:28.872241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:28.872241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:28.872241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:28.872241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:28.872241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:28.872241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:28.872241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:28.872241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:28.873197 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:28.873197 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:28.873197 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:28.873197 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:28.873197 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:28.873197 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:28.873197 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:28.873197 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:28.873197 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:28.873197 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:28.873197 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:28.873197 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:28.873197 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:28.873197 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:28.873197 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:28.873197 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:28.874860 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:28.874860 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:28.874860 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:28.874860 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:28.874860 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:28.874860 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:28.874860 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:28.874860 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:28.874860 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:28.874860 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:28.874860 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 15:33:28.874860 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:28.874860 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 186/359] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:33:28 2022] GET /identity/v3/roles/reader => generated 84 bytes in 252 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:28.877371 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b462ecd-b446-4634-95e5-36e4f782e28b None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:28.877724 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b462ecd-b446-4634-95e5-36e4f782e28b None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:28.879520 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b462ecd-b446-4634-95e5-36e4f782e28b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:28.919300 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-1b462ecd-b446-4634-95e5-36e4f782e28b None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:28.922334 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-1b462ecd-b446-4634-95e5-36e4f782e28b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:29.009932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:29.022316 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 187/360] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:28 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:29.023277 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b462ecd-b446-4634-95e5-36e4f782e28b None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:29.028929 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434ed11c-abfa-4d73-8e8b-5e5b25a85305 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:29.132223 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed020bf6-2b90-4606-8e33-769495c45ed7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:29.133330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed020bf6-2b90-4606-8e33-769495c45ed7 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:29.133812 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed020bf6-2b90-4606-8e33-769495c45ed7 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:29.134283 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed020bf6-2b90-4606-8e33-769495c45ed7 None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:29.135164 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed020bf6-2b90-4606-8e33-769495c45ed7 None admin] RBAC: Adding query filter params (name=member) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:29.135698 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed020bf6-2b90-4606-8e33-769495c45ed7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:29.137299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed020bf6-2b90-4606-8e33-769495c45ed7 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:29.148338 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b462ecd-b446-4634-95e5-36e4f782e28b None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:29.156063 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 174/361] 10.222.0.102 () {56 vars in 1106 bytes} [Wed Aug 24 15:33:28 2022] GET /identity/v3/roles?name=member => generated 343 bytes in 362 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:29.157346 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed020bf6-2b90-4606-8e33-769495c45ed7 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:29.181169 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:29.189892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab7942e-32ed-43ea-99fe-b8ec644a5ace None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:29.190626 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab7942e-32ed-43ea-99fe-b8ec644a5ace None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:29.190905 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab7942e-32ed-43ea-99fe-b8ec644a5ace None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:29.191287 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab7942e-32ed-43ea-99fe-b8ec644a5ace None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:29.192052 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab7942e-32ed-43ea-99fe-b8ec644a5ace None admin] RBAC: Adding query filter params (name=reader) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:29.192358 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab7942e-32ed-43ea-99fe-b8ec644a5ace None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:29.193876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab7942e-32ed-43ea-99fe-b8ec644a5ace None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:29.204414 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 188/362] 10.222.0.102 () {56 vars in 1106 bytes} [Wed Aug 24 15:33:29 2022] GET /identity/v3/roles?name=reader => generated 343 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:29.205003 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab7942e-32ed-43ea-99fe-b8ec644a5ace None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:29.307846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed020bf6-2b90-4606-8e33-769495c45ed7 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:29.319857 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab7942e-32ed-43ea-99fe-b8ec644a5ace None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:29.350614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c19c23-07d3-49d9-832a-d0e82fe80ec3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:29.351587 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c19c23-07d3-49d9-832a-d0e82fe80ec3 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:29.352123 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c19c23-07d3-49d9-832a-d0e82fe80ec3 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:29.352596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c19c23-07d3-49d9-832a-d0e82fe80ec3 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:29.359013 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41f01c7-81a4-4ef2-ae4c-2642dbd19a7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:29.359616 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41f01c7-81a4-4ef2-ae4c-2642dbd19a7a None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:29.359922 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41f01c7-81a4-4ef2-ae4c-2642dbd19a7a None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:29.360193 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41f01c7-81a4-4ef2-ae4c-2642dbd19a7a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:29.368425 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c19c23-07d3-49d9-832a-d0e82fe80ec3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:29.375453 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-20c19c23-07d3-49d9-832a-d0e82fe80ec3 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 15:33:29.375453 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:29.375453 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:29.375453 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:29.375453 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:29.375453 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:29.375453 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:29.375453 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:29.375453 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:29.375453 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:29.375453 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:29.375453 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:29.375453 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:29.375453 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:29.375453 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:29.375453 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:29.376586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:29.376586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:29.376586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:29.376586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:29.376586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:29.376586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:29.376586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:29.376586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:29.376586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:29.376586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:29.376586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:29.376586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:29.376586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:29.376586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:29.376586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:29.376586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:29.377571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:29.377571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:29.377571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:29.377571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:29.377571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:29.377571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:29.377571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:29.377571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:29.377571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:29.377571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:29.377571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 15:33:29.377571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:29.379503 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 175/363] 10.222.0.102 () {56 vars in 1119 bytes} [Wed Aug 24 15:33:29 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:29.380450 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c19c23-07d3-49d9-832a-d0e82fe80ec3 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:29.398883 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:29.428722 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:29.446930 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c41f01c7-81a4-4ef2-ae4c-2642dbd19a7a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:29.448741 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c41f01c7-81a4-4ef2-ae4c-2642dbd19a7a None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:29.495032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c19c23-07d3-49d9-832a-d0e82fe80ec3 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:29.534974 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e45b58-1fe2-4add-bf67-192acba71f32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:29.535640 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e45b58-1fe2-4add-bf67-192acba71f32 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:29.535933 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e45b58-1fe2-4add-bf67-192acba71f32 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:29.536206 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e45b58-1fe2-4add-bf67-192acba71f32 None admin] PATH_INFO: `/v3/users/8368f63e72a24f0f8e110787042d302e` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:29.538105 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e45b58-1fe2-4add-bf67-192acba71f32 None admin] RBAC: Authorizing `identity:get_user(user_id=8368f63e72a24f0f8e110787042d302e)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:29.539580 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e45b58-1fe2-4add-bf67-192acba71f32 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:29.541622 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 176/364] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:33:29 2022] GET /identity/v3/users/8368f63e72a24f0f8e110787042d302e => generated 294 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:29.542410 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e45b58-1fe2-4add-bf67-192acba71f32 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:29.564531 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 189/365] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:33:29 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:33:29.565837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41f01c7-81a4-4ef2-ae4c-2642dbd19a7a None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:29.676067 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:29.681423 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41f01c7-81a4-4ef2-ae4c-2642dbd19a7a None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:29.686265 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:29.714216 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e45b58-1fe2-4add-bf67-192acba71f32 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:29.724830 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0639622b-a1d3-40f1-8006-3caab36c668c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:29.725691 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0639622b-a1d3-40f1-8006-3caab36c668c None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:29.726271 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0639622b-a1d3-40f1-8006-3caab36c668c None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:29.726719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0639622b-a1d3-40f1-8006-3caab36c668c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:29.727607 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0639622b-a1d3-40f1-8006-3caab36c668c None admin] RBAC: Adding query filter params (role.id=868817cab1414b5aa207bc5bdf4f1d9a, user.id=8368f63e72a24f0f8e110787042d302e, scope.system=all) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:29.728195 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0639622b-a1d3-40f1-8006-3caab36c668c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:29.729826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0639622b-a1d3-40f1-8006-3caab36c668c None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:29.742809 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 190/366] 10.222.0.102 () {56 vars in 1313 bytes} [Wed Aug 24 15:33:29 2022] GET /identity/v3/role_assignments?user.id=8368f63e72a24f0f8e110787042d302e&scope.system=all&role.id=868817cab1414b5aa207bc5bdf4f1d9a => generated 512 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:29.744050 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0639622b-a1d3-40f1-8006-3caab36c668c None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:29.773759 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c989a1-8b82-4d0d-84d1-7a56c4304f6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:29.775420 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c989a1-8b82-4d0d-84d1-7a56c4304f6d None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:29.776239 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c989a1-8b82-4d0d-84d1-7a56c4304f6d None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:29.777902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c989a1-8b82-4d0d-84d1-7a56c4304f6d None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:29.779157 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c989a1-8b82-4d0d-84d1-7a56c4304f6d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:29.780804 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c989a1-8b82-4d0d-84d1-7a56c4304f6d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:29.784111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c989a1-8b82-4d0d-84d1-7a56c4304f6d None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:29.805110 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 177/367] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:33:29 2022] GET /identity/v3/roles?name=anotherrole => generated 336 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:29.807417 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c989a1-8b82-4d0d-84d1-7a56c4304f6d None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:29.976871 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0639622b-a1d3-40f1-8006-3caab36c668c None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:30.025493 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c989a1-8b82-4d0d-84d1-7a56c4304f6d None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:30.034476 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58467c5c-023b-444e-8170-70666091ab54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:30.035122 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58467c5c-023b-444e-8170-70666091ab54 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:30.035402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58467c5c-023b-444e-8170-70666091ab54 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:30.035714 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58467c5c-023b-444e-8170-70666091ab54 None admin] PATH_INFO: `/v3/projects/86a6b464bc2d4f72895f92c759e26a36` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:30.037543 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58467c5c-023b-444e-8170-70666091ab54 None admin] RBAC: Authorizing `identity:get_project(project_id=86a6b464bc2d4f72895f92c759e26a36)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:30.039501 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58467c5c-023b-444e-8170-70666091ab54 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:30.042293 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 191/368] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:29 2022] GET /identity/v3/projects/86a6b464bc2d4f72895f92c759e26a36 => generated 303 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:30.043354 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58467c5c-023b-444e-8170-70666091ab54 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:30.069378 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:30.075317 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b97ce9-b645-42e7-b3ec-be0ec9dc2052 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:30.076031 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b97ce9-b645-42e7-b3ec-be0ec9dc2052 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:30.076305 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b97ce9-b645-42e7-b3ec-be0ec9dc2052 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:30.077363 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b97ce9-b645-42e7-b3ec-be0ec9dc2052 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:30.097983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b97ce9-b645-42e7-b3ec-be0ec9dc2052 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:30.103412 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-98b97ce9-b645-42e7-b3ec-be0ec9dc2052 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:33:30.103412 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:30.103412 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:30.103412 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:30.103412 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:30.103412 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:30.103412 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:30.103412 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:30.103412 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:30.103412 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:30.103412 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:30.103412 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:30.103412 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:30.103412 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:30.103412 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:30.103412 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:30.104376 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:30.104376 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:30.104376 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:30.104376 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:30.104376 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:30.104376 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:30.104376 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:30.104376 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:30.104376 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:30.104376 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:30.104376 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:30.104376 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:30.104376 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:30.104376 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:30.104376 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:30.104376 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:30.105151 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:30.105151 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:30.105151 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:30.105151 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:30.105151 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:30.105151 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:30.105151 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:30.105151 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:30.105151 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:30.105151 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:30.105151 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:33:30.105151 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:30.106232 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 178/369] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:33:29 2022] GET /identity/v3/roles/admin => generated 83 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:30.107435 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b97ce9-b645-42e7-b3ec-be0ec9dc2052 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:30.114149 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:30.199103 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58467c5c-023b-444e-8170-70666091ab54 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:30.229800 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b97ce9-b645-42e7-b3ec-be0ec9dc2052 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:30.246597 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ddbbd3-edf3-4289-864c-c8fcb5890fd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:30.247219 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ddbbd3-edf3-4289-864c-c8fcb5890fd6 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:30.247686 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ddbbd3-edf3-4289-864c-c8fcb5890fd6 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:30.248123 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ddbbd3-edf3-4289-864c-c8fcb5890fd6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:30.270739 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e099f2cf-31e4-4c7e-bed2-db5d2757cce6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:30.271385 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e099f2cf-31e4-4c7e-bed2-db5d2757cce6 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:30.271693 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e099f2cf-31e4-4c7e-bed2-db5d2757cce6 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:30.272088 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e099f2cf-31e4-4c7e-bed2-db5d2757cce6 None admin] PATH_INFO: `/v3/groups/6d47b38427704c8baaaefd6a3ae615a6` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:30.273297 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e099f2cf-31e4-4c7e-bed2-db5d2757cce6 None admin] RBAC: Authorizing `identity:get_group(group_id=6d47b38427704c8baaaefd6a3ae615a6)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:30.274793 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e099f2cf-31e4-4c7e-bed2-db5d2757cce6 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:30.276766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 179/370] 10.222.0.102 () {56 vars in 1185 bytes} [Wed Aug 24 15:33:30 2022] GET /identity/v3/groups/6d47b38427704c8baaaefd6a3ae615a6 => generated 227 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:30.277565 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e099f2cf-31e4-4c7e-bed2-db5d2757cce6 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:30.334080 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ddbbd3-edf3-4289-864c-c8fcb5890fd6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:30.335975 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ddbbd3-edf3-4289-864c-c8fcb5890fd6 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:30.358868 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:30.361552 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:30.394299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e099f2cf-31e4-4c7e-bed2-db5d2757cce6 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:30.432223 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f7b003-e2c9-4a73-bf0a-ecef2106db79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:30.432835 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f7b003-e2c9-4a73-bf0a-ecef2106db79 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:30.433115 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f7b003-e2c9-4a73-bf0a-ecef2106db79 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:30.433385 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f7b003-e2c9-4a73-bf0a-ecef2106db79 None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:30.434057 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f7b003-e2c9-4a73-bf0a-ecef2106db79 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:30.434368 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f7b003-e2c9-4a73-bf0a-ecef2106db79 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:30.435814 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f7b003-e2c9-4a73-bf0a-ecef2106db79 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:30.445565 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 180/371] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:33:30 2022] GET /identity/v3/roles?name=admin => generated 341 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:30.446446 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f7b003-e2c9-4a73-bf0a-ecef2106db79 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:30.466579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 192/372] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:33:30 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:33:30.469586 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ddbbd3-edf3-4289-864c-c8fcb5890fd6 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:30.564298 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f7b003-e2c9-4a73-bf0a-ecef2106db79 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:30.602495 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:30.610117 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70080bc-1a5c-4d6d-abdd-3d478ec10bf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:30.611097 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70080bc-1a5c-4d6d-abdd-3d478ec10bf0 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:30.611500 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70080bc-1a5c-4d6d-abdd-3d478ec10bf0 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:30.611904 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70080bc-1a5c-4d6d-abdd-3d478ec10bf0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:30.612740 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70080bc-1a5c-4d6d-abdd-3d478ec10bf0 None admin] RBAC: Adding query filter params (group.id=6d47b38427704c8baaaefd6a3ae615a6, role.id=b5d2ed3e20ab47eeb741b4f43ab2eebb, scope.project.id=86a6b464bc2d4f72895f92c759e26a36) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:30.613156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70080bc-1a5c-4d6d-abdd-3d478ec10bf0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:30.614896 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70080bc-1a5c-4d6d-abdd-3d478ec10bf0 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:30.624059 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 181/373] 10.222.0.102 () {56 vars in 1381 bytes} [Wed Aug 24 15:33:30 2022] GET /identity/v3/role_assignments?group.id=6d47b38427704c8baaaefd6a3ae615a6&scope.project.id=86a6b464bc2d4f72895f92c759e26a36&role.id=b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 261 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:30.625087 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70080bc-1a5c-4d6d-abdd-3d478ec10bf0 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:30.625323 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ddbbd3-edf3-4289-864c-c8fcb5890fd6 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:30.663481 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:30.696836 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c74d38-52c4-49fa-a336-f3a0a1bbc520 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:30.698719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c74d38-52c4-49fa-a336-f3a0a1bbc520 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:30.699233 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c74d38-52c4-49fa-a336-f3a0a1bbc520 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:30.699721 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c74d38-52c4-49fa-a336-f3a0a1bbc520 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:30.825117 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70080bc-1a5c-4d6d-abdd-3d478ec10bf0 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:30.829543 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c74d38-52c4-49fa-a336-f3a0a1bbc520 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:30.831081 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c74d38-52c4-49fa-a336-f3a0a1bbc520 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:30.909722 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9063f29-2760-428b-800f-abc093a15995 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:30.910371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9063f29-2760-428b-800f-abc093a15995 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:30.910653 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9063f29-2760-428b-800f-abc093a15995 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:30.910927 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9063f29-2760-428b-800f-abc093a15995 None admin] PATH_INFO: `/v3/projects/86a6b464bc2d4f72895f92c759e26a36` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:30.912111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9063f29-2760-428b-800f-abc093a15995 None admin] RBAC: Authorizing `identity:get_project(project_id=86a6b464bc2d4f72895f92c759e26a36)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:30.913593 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9063f29-2760-428b-800f-abc093a15995 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:30.915615 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 182/374] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:30 2022] GET /identity/v3/projects/86a6b464bc2d4f72895f92c759e26a36 => generated 303 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:30.926599 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567b3dcb-2a4b-4ffc-9932-0046e67ae81e None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:30.927185 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567b3dcb-2a4b-4ffc-9932-0046e67ae81e None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:30.927660 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567b3dcb-2a4b-4ffc-9932-0046e67ae81e None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:30.929397 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 183/375] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:30 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:33:30.934160 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567b3dcb-2a4b-4ffc-9932-0046e67ae81e None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:31.008472 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 193/376] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:33:30 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 539 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:33:31.009892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279f43d8-76f3-4f00-a003-af05d05e381c None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:31.010163 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279f43d8-76f3-4f00-a003-af05d05e381c None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:31.010446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279f43d8-76f3-4f00-a003-af05d05e381c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:31.033628 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-279f43d8-76f3-4f00-a003-af05d05e381c None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:31.034839 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-279f43d8-76f3-4f00-a003-af05d05e381c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:31.035511 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:31.053606 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:31.138772 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567b3dcb-2a4b-4ffc-9932-0046e67ae81e None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:31.155729 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 194/377] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:31 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:31.155729 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279f43d8-76f3-4f00-a003-af05d05e381c None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:31.203715 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ffd542-a655-4ab0-ae26-0a643bb7e499 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:31.204292 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ffd542-a655-4ab0-ae26-0a643bb7e499 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:31.204567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ffd542-a655-4ab0-ae26-0a643bb7e499 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:31.204945 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ffd542-a655-4ab0-ae26-0a643bb7e499 None admin] PATH_INFO: `/v3/groups/6d47b38427704c8baaaefd6a3ae615a6` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:31.206231 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ffd542-a655-4ab0-ae26-0a643bb7e499 None admin] RBAC: Authorizing `identity:get_group(group_id=6d47b38427704c8baaaefd6a3ae615a6)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:31.207689 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ffd542-a655-4ab0-ae26-0a643bb7e499 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:31.209308 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 184/378] 10.222.0.102 () {56 vars in 1185 bytes} [Wed Aug 24 15:33:30 2022] GET /identity/v3/groups/6d47b38427704c8baaaefd6a3ae615a6 => generated 227 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:31.216581 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ffd542-a655-4ab0-ae26-0a643bb7e499 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:31.347754 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279f43d8-76f3-4f00-a003-af05d05e381c None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:31.397247 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ffd542-a655-4ab0-ae26-0a643bb7e499 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:31.417268 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8d134a-c2c3-4e26-bed8-737560d90d61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:31.417983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8d134a-c2c3-4e26-bed8-737560d90d61 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:31.418267 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8d134a-c2c3-4e26-bed8-737560d90d61 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:31.418591 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8d134a-c2c3-4e26-bed8-737560d90d61 None admin] PATH_INFO: `/v3/projects/501a315437844349bdc02dafe337b855` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:31.420549 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c8d134a-c2c3-4e26-bed8-737560d90d61 None admin] RBAC: Authorizing `identity:get_project(project_id=501a315437844349bdc02dafe337b855)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:31.422266 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c8d134a-c2c3-4e26-bed8-737560d90d61 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:31.430319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 195/379] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:31 2022] GET /identity/v3/projects/501a315437844349bdc02dafe337b855 => generated 345 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:31.430319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8d134a-c2c3-4e26-bed8-737560d90d61 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:31.441135 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:31.452698 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8324d72-971a-4d58-8401-37999a95cde8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:31.454043 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8324d72-971a-4d58-8401-37999a95cde8 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:31.454658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8324d72-971a-4d58-8401-37999a95cde8 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:31.455265 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8324d72-971a-4d58-8401-37999a95cde8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:31.456242 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8324d72-971a-4d58-8401-37999a95cde8 None admin] RBAC: Adding query filter params (group.id=6d47b38427704c8baaaefd6a3ae615a6, role.id=b8429ebd9c2a4cbbbbf30ad56ce23bfa, scope.project.id=86a6b464bc2d4f72895f92c759e26a36) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:31.456759 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8324d72-971a-4d58-8401-37999a95cde8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:31.458555 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8324d72-971a-4d58-8401-37999a95cde8 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:31.465406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 185/380] 10.222.0.102 () {56 vars in 1381 bytes} [Wed Aug 24 15:33:31 2022] GET /identity/v3/role_assignments?group.id=6d47b38427704c8baaaefd6a3ae615a6&scope.project.id=86a6b464bc2d4f72895f92c759e26a36&role.id=b8429ebd9c2a4cbbbbf30ad56ce23bfa => generated 261 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:31.468128 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:31.518497 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61328088-12f8-4dbd-80a2-9d42180772da None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:31.519170 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61328088-12f8-4dbd-80a2-9d42180772da None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:31.519312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61328088-12f8-4dbd-80a2-9d42180772da None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:31.521128 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 186/381] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:31 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:33:31.526788 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a535b3-629b-4fe3-8f17-61809d0a388d None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:31.527207 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a535b3-629b-4fe3-8f17-61809d0a388d None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:31.527607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a535b3-629b-4fe3-8f17-61809d0a388d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:31.548407 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-65a535b3-629b-4fe3-8f17-61809d0a388d None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:31.549805 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-65a535b3-629b-4fe3-8f17-61809d0a388d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:31.644477 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8d134a-c2c3-4e26-bed8-737560d90d61 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:31.653846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 187/382] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:31 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:31.655367 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a535b3-629b-4fe3-8f17-61809d0a388d None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:31.697111 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07193a29-9be9-4862-8891-8b3399f71737 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:31.698007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07193a29-9be9-4862-8891-8b3399f71737 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:31.699706 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07193a29-9be9-4862-8891-8b3399f71737 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:31.700153 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07193a29-9be9-4862-8891-8b3399f71737 None admin] PATH_INFO: `/v3/groups/fb2a9e838fbd4936b9e7968ffe35e3eb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:31.715285 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07193a29-9be9-4862-8891-8b3399f71737 None admin] RBAC: Authorizing `identity:get_group(group_id=fb2a9e838fbd4936b9e7968ffe35e3eb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:31.722012 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07193a29-9be9-4862-8891-8b3399f71737 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:31.724316 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 196/383] 10.222.0.102 () {56 vars in 1185 bytes} [Wed Aug 24 15:33:31 2022] GET /identity/v3/groups/fb2a9e838fbd4936b9e7968ffe35e3eb => generated 230 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:33:31.729698 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07193a29-9be9-4862-8891-8b3399f71737 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:31.852075 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:31.854530 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a535b3-629b-4fe3-8f17-61809d0a388d None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:31.872190 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07193a29-9be9-4862-8891-8b3399f71737 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:31.895455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:31.912235 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629425f8-24ef-435d-bd03-5dec1ae9b61f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:31.913628 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629425f8-24ef-435d-bd03-5dec1ae9b61f None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:31.914556 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629425f8-24ef-435d-bd03-5dec1ae9b61f None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:31.915287 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629425f8-24ef-435d-bd03-5dec1ae9b61f None admin] PATH_INFO: `/v3/roles/member` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:31.917116 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67915f41-d880-4422-93e6-de912be09ab6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:31.917921 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67915f41-d880-4422-93e6-de912be09ab6 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:31.918515 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67915f41-d880-4422-93e6-de912be09ab6 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:31.924111 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67915f41-d880-4422-93e6-de912be09ab6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:31.924926 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67915f41-d880-4422-93e6-de912be09ab6 None admin] RBAC: Adding query filter params (group.id=fb2a9e838fbd4936b9e7968ffe35e3eb, role.id=4d8fbbaea81a4145b89af12350bb067c, scope.project.id=501a315437844349bdc02dafe337b855) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:31.925248 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67915f41-d880-4422-93e6-de912be09ab6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:31.927301 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67915f41-d880-4422-93e6-de912be09ab6 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:31.931568 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629425f8-24ef-435d-bd03-5dec1ae9b61f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:31.938036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-629425f8-24ef-435d-bd03-5dec1ae9b61f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:33:31.938036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:31.938036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:31.938036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:31.938036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:31.938036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:31.938036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:31.938036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:31.938036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:31.938036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:31.938036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:31.938036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:31.938036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:31.938036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:31.938036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:31.938036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:31.939155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:31.939155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:31.939155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:31.939155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:31.939155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:31.939155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:31.939155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:31.939155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:31.939155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:31.939155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:31.939155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:31.939155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:31.939155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:31.939155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:31.939155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:31.939155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:31.940146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:31.940146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:31.940146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:31.940146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:31.940146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:31.940146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:31.940146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:31.940146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:31.940146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:31.940146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:31.940146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:33:31.940146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:31.944312 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 197/385] 10.222.0.102 () {56 vars in 1381 bytes} [Wed Aug 24 15:33:31 2022] GET /identity/v3/role_assignments?group.id=fb2a9e838fbd4936b9e7968ffe35e3eb&scope.project.id=501a315437844349bdc02dafe337b855&role.id=4d8fbbaea81a4145b89af12350bb067c => generated 261 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:33:31.944554 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 188/385] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:33:31 2022] GET /identity/v3/roles/member => generated 84 bytes in 288 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:31.953624 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67915f41-d880-4422-93e6-de912be09ab6 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:32.041156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629425f8-24ef-435d-bd03-5dec1ae9b61f None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:32.126115 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67915f41-d880-4422-93e6-de912be09ab6 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:32.185260 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d36aea-fbce-41a1-b237-32a2b33d2504 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:32.187143 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d36aea-fbce-41a1-b237-32a2b33d2504 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:32.187581 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d36aea-fbce-41a1-b237-32a2b33d2504 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:32.188020 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d36aea-fbce-41a1-b237-32a2b33d2504 None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:32.188844 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21d36aea-fbce-41a1-b237-32a2b33d2504 None admin] RBAC: Adding query filter params (name=member) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:32.189316 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21d36aea-fbce-41a1-b237-32a2b33d2504 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:32.190882 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21d36aea-fbce-41a1-b237-32a2b33d2504 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:32.208060 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 198/386] 10.222.0.102 () {56 vars in 1106 bytes} [Wed Aug 24 15:33:31 2022] GET /identity/v3/roles?name=member => generated 343 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:32.209946 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d36aea-fbce-41a1-b237-32a2b33d2504 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:32.219560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629425f8-24ef-435d-bd03-5dec1ae9b61f None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:32.223115 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:32.264744 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4a6bbf-7eb0-4fe2-a658-ad3550a6916f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:32.265386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4a6bbf-7eb0-4fe2-a658-ad3550a6916f None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:32.265664 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4a6bbf-7eb0-4fe2-a658-ad3550a6916f None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:32.265936 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4a6bbf-7eb0-4fe2-a658-ad3550a6916f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:32.283172 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4a6bbf-7eb0-4fe2-a658-ad3550a6916f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:32.288754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-6c4a6bbf-7eb0-4fe2-a658-ad3550a6916f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 15:33:32.288754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:32.288754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:32.288754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:32.288754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:32.288754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:32.288754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:32.288754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:32.288754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:32.288754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:32.288754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:32.288754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:32.288754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:32.288754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:32.288754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:32.288754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:32.289959 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:32.289959 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:32.289959 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:32.289959 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:32.289959 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:32.289959 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:32.289959 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:32.289959 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:32.289959 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:32.289959 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:32.289959 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:32.289959 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:32.289959 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:32.289959 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:32.289959 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:32.289959 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:32.291081 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:32.291081 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:32.291081 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:32.291081 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:32.291081 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:32.291081 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:32.291081 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:32.291081 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:32.291081 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:32.291081 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:32.291081 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 15:33:32.291081 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:32.291081 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 189/387] 10.222.0.102 () {56 vars in 1119 bytes} [Wed Aug 24 15:33:32 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:33:32.298255 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4a6bbf-7eb0-4fe2-a658-ad3550a6916f None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:32.334696 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d36aea-fbce-41a1-b237-32a2b33d2504 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:32.357418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:32.373789 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696cf488-3fab-4761-9298-a1f9fa8fb0af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:32.374907 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696cf488-3fab-4761-9298-a1f9fa8fb0af None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:32.375386 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696cf488-3fab-4761-9298-a1f9fa8fb0af None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:32.375834 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696cf488-3fab-4761-9298-a1f9fa8fb0af None admin] PATH_INFO: `/v3/groups/6d47b38427704c8baaaefd6a3ae615a6` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:32.377175 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696cf488-3fab-4761-9298-a1f9fa8fb0af None admin] RBAC: Authorizing `identity:get_group(group_id=6d47b38427704c8baaaefd6a3ae615a6)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:32.379021 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696cf488-3fab-4761-9298-a1f9fa8fb0af None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:32.385483 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 199/388] 10.222.0.102 () {56 vars in 1185 bytes} [Wed Aug 24 15:33:32 2022] GET /identity/v3/groups/6d47b38427704c8baaaefd6a3ae615a6 => generated 227 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:33:32.387182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696cf488-3fab-4761-9298-a1f9fa8fb0af None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:32.422458 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4a6bbf-7eb0-4fe2-a658-ad3550a6916f None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:32.462106 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0127f2-9798-4a60-b9a4-0b35be86868d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:32.462348 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0127f2-9798-4a60-b9a4-0b35be86868d None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:32.462592 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0127f2-9798-4a60-b9a4-0b35be86868d None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:32.462864 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0127f2-9798-4a60-b9a4-0b35be86868d None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:32.463532 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a0127f2-9798-4a60-b9a4-0b35be86868d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:32.463887 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a0127f2-9798-4a60-b9a4-0b35be86868d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:32.465248 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a0127f2-9798-4a60-b9a4-0b35be86868d None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:32.473473 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:32.476885 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 190/389] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:33:32 2022] GET /identity/v3/roles?name=anotherrole => generated 336 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:33:32.481751 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0127f2-9798-4a60-b9a4-0b35be86868d None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:32.503239 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696cf488-3fab-4761-9298-a1f9fa8fb0af None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:32.542631 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dcbbfa4-4b30-4bd5-b124-d81a6b7e900d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:32.543547 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcbbfa4-4b30-4bd5-b124-d81a6b7e900d None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:32.543997 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcbbfa4-4b30-4bd5-b124-d81a6b7e900d None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:32.544414 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcbbfa4-4b30-4bd5-b124-d81a6b7e900d None admin] PATH_INFO: `/v3/projects/749c6d651fc445cdb75db25a73873386` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:32.546196 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dcbbfa4-4b30-4bd5-b124-d81a6b7e900d None admin] RBAC: Authorizing `identity:get_project(project_id=749c6d651fc445cdb75db25a73873386)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:32.547861 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dcbbfa4-4b30-4bd5-b124-d81a6b7e900d None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:32.554739 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 200/390] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:32 2022] GET /identity/v3/projects/749c6d651fc445cdb75db25a73873386 => generated 299 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:32.556175 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dcbbfa4-4b30-4bd5-b124-d81a6b7e900d None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:32.627655 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0127f2-9798-4a60-b9a4-0b35be86868d None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:32.637317 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:32.682615 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b17a22-b26b-45a8-8b13-f2d67ff9aab0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:32.683259 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b17a22-b26b-45a8-8b13-f2d67ff9aab0 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:32.683537 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b17a22-b26b-45a8-8b13-f2d67ff9aab0 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:32.683835 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b17a22-b26b-45a8-8b13-f2d67ff9aab0 None admin] PATH_INFO: `/v3/groups/6d47b38427704c8baaaefd6a3ae615a6` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:32.691759 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b17a22-b26b-45a8-8b13-f2d67ff9aab0 None admin] RBAC: Authorizing `identity:get_group(group_id=6d47b38427704c8baaaefd6a3ae615a6)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:32.693434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b17a22-b26b-45a8-8b13-f2d67ff9aab0 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:32.695369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 191/391] 10.222.0.102 () {56 vars in 1185 bytes} [Wed Aug 24 15:33:32 2022] GET /identity/v3/groups/6d47b38427704c8baaaefd6a3ae615a6 => generated 227 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:32.701051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b17a22-b26b-45a8-8b13-f2d67ff9aab0 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:32.707949 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dcbbfa4-4b30-4bd5-b124-d81a6b7e900d None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:32.776374 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4406853-d617-4766-a9b7-30ec30612a06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:32.777065 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4406853-d617-4766-a9b7-30ec30612a06 None admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:32.777351 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4406853-d617-4766-a9b7-30ec30612a06 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:32.777633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4406853-d617-4766-a9b7-30ec30612a06 None admin] PATH_INFO: `/v3/projects/749c6d651fc445cdb75db25a73873386/groups/6d47b38427704c8baaaefd6a3ae615a6/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:32.783261 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4406853-d617-4766-a9b7-30ec30612a06 None admin] RBAC: Authorizing `identity:create_grant(project_id=749c6d651fc445cdb75db25a73873386, group_id=6d47b38427704c8baaaefd6a3ae615a6, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:32.784764 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4406853-d617-4766-a9b7-30ec30612a06 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:32.824185 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 201/392] 10.222.0.102 () {58 vars in 1447 bytes} [Wed Aug 24 15:33:32 2022] PUT /identity/v3/projects/749c6d651fc445cdb75db25a73873386/groups/6d47b38427704c8baaaefd6a3ae615a6/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:33:32.872379 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b17a22-b26b-45a8-8b13-f2d67ff9aab0 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:32.928448 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76e2f3c-29f7-4c35-a5e5-245686f2b59e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:32.930103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76e2f3c-29f7-4c35-a5e5-245686f2b59e None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:32.930943 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76e2f3c-29f7-4c35-a5e5-245686f2b59e None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:32.931668 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76e2f3c-29f7-4c35-a5e5-245686f2b59e None admin] PATH_INFO: `/v3/projects/749c6d651fc445cdb75db25a73873386` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:32.934278 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76e2f3c-29f7-4c35-a5e5-245686f2b59e None admin] RBAC: Authorizing `identity:get_project(project_id=749c6d651fc445cdb75db25a73873386)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:32.937648 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76e2f3c-29f7-4c35-a5e5-245686f2b59e None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:32.940769 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 192/393] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:32 2022] GET /identity/v3/projects/749c6d651fc445cdb75db25a73873386 => generated 299 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:33:32.948856 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4406853-d617-4766-a9b7-30ec30612a06 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:32.955773 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:33.103415 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4406853-d617-4766-a9b7-30ec30612a06 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:33.146539 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f733db42-902b-4ec6-83c9-613b56a839da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:33.147296 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f733db42-902b-4ec6-83c9-613b56a839da None admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:33.147547 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f733db42-902b-4ec6-83c9-613b56a839da None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:33.147853 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f733db42-902b-4ec6-83c9-613b56a839da None admin] PATH_INFO: `/v3/projects/749c6d651fc445cdb75db25a73873386/groups/6d47b38427704c8baaaefd6a3ae615a6/roles/b8429ebd9c2a4cbbbbf30ad56ce23bfa` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:33.150051 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f733db42-902b-4ec6-83c9-613b56a839da None admin] RBAC: Authorizing `identity:create_grant(project_id=749c6d651fc445cdb75db25a73873386, group_id=6d47b38427704c8baaaefd6a3ae615a6, role_id=b8429ebd9c2a4cbbbbf30ad56ce23bfa)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:33.151570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f733db42-902b-4ec6-83c9-613b56a839da None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:33.166373 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 202/394] 10.222.0.102 () {58 vars in 1447 bytes} [Wed Aug 24 15:33:32 2022] PUT /identity/v3/projects/749c6d651fc445cdb75db25a73873386/groups/6d47b38427704c8baaaefd6a3ae615a6/roles/b8429ebd9c2a4cbbbbf30ad56ce23bfa => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:33:33.328256 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ec73e4-9b8e-4763-9d22-5f0c1d4ab9fa None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:33.328775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ec73e4-9b8e-4763-9d22-5f0c1d4ab9fa None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:33.329109 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ec73e4-9b8e-4763-9d22-5f0c1d4ab9fa None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:33.330998 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 193/395] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:33 2022] GET /identity => generated 271 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:33.336391 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a742255-42ae-4273-8b42-a9cc39e78053 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:33.336680 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a742255-42ae-4273-8b42-a9cc39e78053 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:33.336959 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a742255-42ae-4273-8b42-a9cc39e78053 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:33.359621 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-8a742255-42ae-4273-8b42-a9cc39e78053 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:33.360781 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-8a742255-42ae-4273-8b42-a9cc39e78053 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:33.444059 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:33.456012 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 203/396] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:33 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:33.781061 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ec73e4-9b8e-4763-9d22-5f0c1d4ab9fa None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:33.867653 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:33.909105 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14cdadb-9093-472f-9dc5-f36057484e0c None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:33.909442 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14cdadb-9093-472f-9dc5-f36057484e0c None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:33.909667 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14cdadb-9093-472f-9dc5-f36057484e0c None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:33.911473 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 204/397] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:33 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:33.919110 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbb55fb-3057-4d3b-935a-873370d8d73a None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:33.919570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbb55fb-3057-4d3b-935a-873370d8d73a None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:33.920017 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbb55fb-3057-4d3b-935a-873370d8d73a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:33.955927 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ec73e4-9b8e-4763-9d22-5f0c1d4ab9fa None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:33.974024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-3fbb55fb-3057-4d3b-935a-873370d8d73a None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:33.975242 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-3fbb55fb-3057-4d3b-935a-873370d8d73a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:34.036550 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf55714-9cac-4dc4-99a4-8b23cc85cb4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:34.037125 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf55714-9cac-4dc4-99a4-8b23cc85cb4c None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:34.037405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf55714-9cac-4dc4-99a4-8b23cc85cb4c None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:34.037676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf55714-9cac-4dc4-99a4-8b23cc85cb4c None admin] PATH_INFO: `/v3/roles/member` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:34.061155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bf55714-9cac-4dc4-99a4-8b23cc85cb4c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:34.066684 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-2bf55714-9cac-4dc4-99a4-8b23cc85cb4c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:33:34.066684 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:34.066684 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:34.066684 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:34.066684 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:34.066684 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:34.066684 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:34.066684 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:34.066684 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:34.066684 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:34.066684 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:34.066684 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:34.066684 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:34.066684 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:34.066684 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:34.066684 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:34.067586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:34.067586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:34.067586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:34.067586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:34.067586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:34.067586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:34.067586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:34.067586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:34.067586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:34.067586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:34.067586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:34.067586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:34.067586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:34.067586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:34.067586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:34.067586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:34.069620 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:34.069620 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:34.069620 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:34.069620 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:34.069620 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:34.069620 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:34.069620 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:34.069620 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:34.069620 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:34.069620 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:34.069620 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:33:34.069620 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:34.069620 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 194/398] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:33:33 2022] GET /identity/v3/roles/member => generated 84 bytes in 288 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:34.078688 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf55714-9cac-4dc4-99a4-8b23cc85cb4c None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:34.118408 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 205/399] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:33 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:34.250582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf55714-9cac-4dc4-99a4-8b23cc85cb4c None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:34.256706 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:34.302329 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4085ea02-9c66-4134-9b05-33fd6bce040f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:34.302867 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4085ea02-9c66-4134-9b05-33fd6bce040f None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:34.303149 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4085ea02-9c66-4134-9b05-33fd6bce040f None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:34.303425 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4085ea02-9c66-4134-9b05-33fd6bce040f None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:34.304134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4085ea02-9c66-4134-9b05-33fd6bce040f None admin] RBAC: Adding query filter params (name=member) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:34.304438 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4085ea02-9c66-4134-9b05-33fd6bce040f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:34.305826 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4085ea02-9c66-4134-9b05-33fd6bce040f None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:34.320339 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 195/400] 10.222.0.102 () {56 vars in 1106 bytes} [Wed Aug 24 15:33:34 2022] GET /identity/v3/roles?name=member => generated 343 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:34.325727 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fbb55fb-3057-4d3b-935a-873370d8d73a None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:34.449042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4085ea02-9c66-4134-9b05-33fd6bce040f None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:34.458804 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fbb55fb-3057-4d3b-935a-873370d8d73a None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:34.507700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:34.534978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec95e52a-bdd3-49c5-9b4a-c110e0b1af26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:34.535664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec95e52a-bdd3-49c5-9b4a-c110e0b1af26 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:34.535953 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec95e52a-bdd3-49c5-9b4a-c110e0b1af26 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:34.536229 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec95e52a-bdd3-49c5-9b4a-c110e0b1af26 None admin] PATH_INFO: `/v3/groups/6d47b38427704c8baaaefd6a3ae615a6` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:34.540928 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec95e52a-bdd3-49c5-9b4a-c110e0b1af26 None admin] RBAC: Authorizing `identity:get_group(group_id=6d47b38427704c8baaaefd6a3ae615a6)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:34.542400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec95e52a-bdd3-49c5-9b4a-c110e0b1af26 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:34.544588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 206/401] 10.222.0.102 () {56 vars in 1185 bytes} [Wed Aug 24 15:33:34 2022] GET /identity/v3/groups/6d47b38427704c8baaaefd6a3ae615a6 => generated 227 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:34.550186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec95e52a-bdd3-49c5-9b4a-c110e0b1af26 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:34.655413 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4085ea02-9c66-4134-9b05-33fd6bce040f None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:34.706279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec95e52a-bdd3-49c5-9b4a-c110e0b1af26 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:34.731822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155b1074-8cc6-416c-bbd6-437e1129392a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:34.733666 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155b1074-8cc6-416c-bbd6-437e1129392a None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:34.734158 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155b1074-8cc6-416c-bbd6-437e1129392a None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:34.734644 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155b1074-8cc6-416c-bbd6-437e1129392a None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:34.748705 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39accef2-ccd9-49fa-953f-72c9847dfcf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:34.749686 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39accef2-ccd9-49fa-953f-72c9847dfcf6 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:34.750135 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39accef2-ccd9-49fa-953f-72c9847dfcf6 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:34.750565 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39accef2-ccd9-49fa-953f-72c9847dfcf6 None admin] PATH_INFO: `/v3/projects/86a6b464bc2d4f72895f92c759e26a36` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:34.752028 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39accef2-ccd9-49fa-953f-72c9847dfcf6 None admin] RBAC: Authorizing `identity:get_project(project_id=86a6b464bc2d4f72895f92c759e26a36)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:34.753750 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39accef2-ccd9-49fa-953f-72c9847dfcf6 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:34.756090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 207/402] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:34 2022] GET /identity/v3/projects/86a6b464bc2d4f72895f92c759e26a36 => generated 303 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:34.759659 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155b1074-8cc6-416c-bbd6-437e1129392a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:34.762082 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39accef2-ccd9-49fa-953f-72c9847dfcf6 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:34.765238 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-155b1074-8cc6-416c-bbd6-437e1129392a None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 15:33:34.765238 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:34.765238 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:34.765238 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:34.765238 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:34.765238 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:34.765238 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:34.765238 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:34.765238 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:34.765238 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:34.765238 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:34.765238 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:34.765238 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:34.765238 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:34.765238 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:34.765238 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:34.766108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:34.766108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:34.766108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:34.766108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:34.766108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:34.766108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:34.766108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:34.766108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:34.766108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:34.766108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:34.766108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:34.766108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:34.766108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:34.766108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:34.766108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:34.766108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:34.766891 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:34.766891 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:34.766891 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:34.766891 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:34.766891 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:34.766891 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:34.766891 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:34.766891 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:34.766891 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:34.766891 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:34.766891 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 15:33:34.766891 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:34.766891 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 196/403] 10.222.0.102 () {56 vars in 1119 bytes} [Wed Aug 24 15:33:34 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 318 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:33:34.773189 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155b1074-8cc6-416c-bbd6-437e1129392a None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:34.826940 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:34.844005 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:34.907245 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39accef2-ccd9-49fa-953f-72c9847dfcf6 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:34.931375 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155b1074-8cc6-416c-bbd6-437e1129392a None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:34.984562 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d12d23f-c615-4b6c-83af-1db682a4a97c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:34.985244 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d12d23f-c615-4b6c-83af-1db682a4a97c None admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:34.985721 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d12d23f-c615-4b6c-83af-1db682a4a97c None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:34.986290 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d12d23f-c615-4b6c-83af-1db682a4a97c None admin] PATH_INFO: `/v3/projects/86a6b464bc2d4f72895f92c759e26a36/groups/6d47b38427704c8baaaefd6a3ae615a6/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:34.988949 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d12d23f-c615-4b6c-83af-1db682a4a97c None admin] RBAC: Authorizing `identity:create_grant(project_id=86a6b464bc2d4f72895f92c759e26a36, group_id=6d47b38427704c8baaaefd6a3ae615a6, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:34.989625 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4e62fb-89c5-4aca-ba72-3ee81c3edaf9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:34.990363 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4e62fb-89c5-4aca-ba72-3ee81c3edaf9 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:34.990759 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4e62fb-89c5-4aca-ba72-3ee81c3edaf9 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:34.991084 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d12d23f-c615-4b6c-83af-1db682a4a97c None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:34.991193 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4e62fb-89c5-4aca-ba72-3ee81c3edaf9 None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:34.991991 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee4e62fb-89c5-4aca-ba72-3ee81c3edaf9 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:34.992393 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee4e62fb-89c5-4aca-ba72-3ee81c3edaf9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:34.993985 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee4e62fb-89c5-4aca-ba72-3ee81c3edaf9 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:35.009346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 208/405] 10.222.0.102 () {58 vars in 1447 bytes} [Wed Aug 24 15:33:34 2022] PUT /identity/v3/projects/86a6b464bc2d4f72895f92c759e26a36/groups/6d47b38427704c8baaaefd6a3ae615a6/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 15:33:35.009538 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 197/405] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:33:34 2022] GET /identity/v3/roles?name=anotherrole => generated 336 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:35.015856 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3b7851-5e5f-4f11-b807-d65273e3a7ef None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:35.015856 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3b7851-5e5f-4f11-b807-d65273e3a7ef None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:35.015856 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3b7851-5e5f-4f11-b807-d65273e3a7ef None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:35.015856 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 198/406] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:35 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:35.016499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d12d23f-c615-4b6c-83af-1db682a4a97c None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:35.019666 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08db45c6-6557-4caf-871b-a330b0de128c None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:35.020026 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08db45c6-6557-4caf-871b-a330b0de128c None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:35.020298 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08db45c6-6557-4caf-871b-a330b0de128c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:35.054570 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-08db45c6-6557-4caf-871b-a330b0de128c None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:35.055820 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-08db45c6-6557-4caf-871b-a330b0de128c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:35.204753 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 199/407] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:35 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 15:33:35.217603 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d12d23f-c615-4b6c-83af-1db682a4a97c None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:35.238685 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:35.257355 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707c7f2d-c301-4054-abcc-38e577e61496 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:35.258001 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707c7f2d-c301-4054-abcc-38e577e61496 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:35.258281 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707c7f2d-c301-4054-abcc-38e577e61496 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:35.258552 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707c7f2d-c301-4054-abcc-38e577e61496 None admin] PATH_INFO: `/v3/groups/6d47b38427704c8baaaefd6a3ae615a6` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:35.259687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707c7f2d-c301-4054-abcc-38e577e61496 None admin] RBAC: Authorizing `identity:get_group(group_id=6d47b38427704c8baaaefd6a3ae615a6)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:35.261196 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707c7f2d-c301-4054-abcc-38e577e61496 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:35.263247 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 209/408] 10.222.0.102 () {56 vars in 1185 bytes} [Wed Aug 24 15:33:35 2022] GET /identity/v3/groups/6d47b38427704c8baaaefd6a3ae615a6 => generated 227 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:33:35.268871 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08db45c6-6557-4caf-871b-a330b0de128c None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:35.293417 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:35.385824 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08db45c6-6557-4caf-871b-a330b0de128c None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:35.429644 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659f2937-fb30-415c-b691-c3f627a6bf3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:35.430652 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659f2937-fb30-415c-b691-c3f627a6bf3a None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:35.431151 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659f2937-fb30-415c-b691-c3f627a6bf3a None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:35.431617 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659f2937-fb30-415c-b691-c3f627a6bf3a None admin] PATH_INFO: `/v3/projects/86a6b464bc2d4f72895f92c759e26a36` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:35.434482 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659f2937-fb30-415c-b691-c3f627a6bf3a None admin] RBAC: Authorizing `identity:get_project(project_id=86a6b464bc2d4f72895f92c759e26a36)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:35.436986 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659f2937-fb30-415c-b691-c3f627a6bf3a None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:35.439212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 200/409] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:35 2022] GET /identity/v3/projects/86a6b464bc2d4f72895f92c759e26a36 => generated 303 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:35.445253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707c7f2d-c301-4054-abcc-38e577e61496 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:35.555695 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707c7f2d-c301-4054-abcc-38e577e61496 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:35.566334 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659f2937-fb30-415c-b691-c3f627a6bf3a None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:35.592977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c314b5cb-8032-4c01-984d-a73b55823eb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:35.593723 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c314b5cb-8032-4c01-984d-a73b55823eb7 None admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:35.594024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c314b5cb-8032-4c01-984d-a73b55823eb7 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:35.594285 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c314b5cb-8032-4c01-984d-a73b55823eb7 None admin] PATH_INFO: `/v3/projects/86a6b464bc2d4f72895f92c759e26a36/groups/6d47b38427704c8baaaefd6a3ae615a6/roles/b8429ebd9c2a4cbbbbf30ad56ce23bfa` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:35.596577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c314b5cb-8032-4c01-984d-a73b55823eb7 None admin] RBAC: Authorizing `identity:create_grant(project_id=86a6b464bc2d4f72895f92c759e26a36, group_id=6d47b38427704c8baaaefd6a3ae615a6, role_id=b8429ebd9c2a4cbbbbf30ad56ce23bfa)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:35.598218 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c314b5cb-8032-4c01-984d-a73b55823eb7 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:35.618130 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 210/410] 10.222.0.102 () {58 vars in 1447 bytes} [Wed Aug 24 15:33:35 2022] PUT /identity/v3/projects/86a6b464bc2d4f72895f92c759e26a36/groups/6d47b38427704c8baaaefd6a3ae615a6/roles/b8429ebd9c2a4cbbbbf30ad56ce23bfa => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:33:35.620986 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9873d2-8d54-40b1-95fa-94bb439584bf None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:35.621273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9873d2-8d54-40b1-95fa-94bb439584bf None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:35.621545 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9873d2-8d54-40b1-95fa-94bb439584bf None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:35.623117 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 211/411] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:35 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:35.628499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e401cca3-78c1-4a66-bdc0-b2c6eb19eed2 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:35.628793 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e401cca3-78c1-4a66-bdc0-b2c6eb19eed2 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:35.629051 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e401cca3-78c1-4a66-bdc0-b2c6eb19eed2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:35.672956 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-e401cca3-78c1-4a66-bdc0-b2c6eb19eed2 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:35.675867 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-e401cca3-78c1-4a66-bdc0-b2c6eb19eed2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:35.746229 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:35.771071 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659f2937-fb30-415c-b691-c3f627a6bf3a None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:35.790311 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:35.834465 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 212/412] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:35 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 15:33:35.835680 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0fd5b3-cf34-46fe-9c2b-188bd4ffabef None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:35.835815 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0fd5b3-cf34-46fe-9c2b-188bd4ffabef None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:35.836020 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0fd5b3-cf34-46fe-9c2b-188bd4ffabef None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:35.837206 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48070b07-b741-483d-89b2-b8bb9a895263 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:35.838435 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48070b07-b741-483d-89b2-b8bb9a895263 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:35.838435 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48070b07-b741-483d-89b2-b8bb9a895263 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:35.838435 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48070b07-b741-483d-89b2-b8bb9a895263 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:35.838657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 213/413] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:35 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:35.846751 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625cfcbc-07cb-40f4-b3d1-d8fac50e49c1 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:35.847434 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625cfcbc-07cb-40f4-b3d1-d8fac50e49c1 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:35.847434 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625cfcbc-07cb-40f4-b3d1-d8fac50e49c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:35.858742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48070b07-b741-483d-89b2-b8bb9a895263 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:35.864321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-48070b07-b741-483d-89b2-b8bb9a895263 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:33:35.864321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:35.864321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:35.864321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:35.864321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:35.864321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:35.864321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:35.864321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:35.864321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:35.864321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:35.864321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:35.864321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:35.864321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:35.864321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:35.864321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:35.864321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:35.865220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:35.865220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:35.865220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:35.865220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:35.865220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:35.865220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:35.865220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:35.865220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:35.865220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:35.865220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:35.865220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:35.865220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:35.865220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:35.865220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:35.865220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:35.865220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:35.866917 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:35.866917 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:35.866917 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:35.866917 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:35.866917 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:35.866917 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:35.866917 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:35.866917 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:35.866917 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:35.866917 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:35.866917 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:33:35.866917 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:35.866917 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 201/414] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:33:35 2022] GET /identity/v3/roles/admin => generated 83 bytes in 300 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:35.871692 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48070b07-b741-483d-89b2-b8bb9a895263 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:35.882694 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-625cfcbc-07cb-40f4-b3d1-d8fac50e49c1 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:35.888294 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-625cfcbc-07cb-40f4-b3d1-d8fac50e49c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:36.036403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 214/415] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:35 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:36.077293 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48070b07-b741-483d-89b2-b8bb9a895263 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:36.118296 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db37868-b100-4086-9a8f-f4ab67f3a6cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:36.118878 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db37868-b100-4086-9a8f-f4ab67f3a6cf None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:36.119162 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db37868-b100-4086-9a8f-f4ab67f3a6cf None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:36.119440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db37868-b100-4086-9a8f-f4ab67f3a6cf None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:36.120143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db37868-b100-4086-9a8f-f4ab67f3a6cf None admin] RBAC: Adding query filter params (name=admin) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:36.120451 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db37868-b100-4086-9a8f-f4ab67f3a6cf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:36.121829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db37868-b100-4086-9a8f-f4ab67f3a6cf None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:36.131379 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 202/416] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:33:35 2022] GET /identity/v3/roles?name=admin => generated 341 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:33:36.136663 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625cfcbc-07cb-40f4-b3d1-d8fac50e49c1 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:36.257678 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db37868-b100-4086-9a8f-f4ab67f3a6cf None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:36.271465 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:36.289472 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625cfcbc-07cb-40f4-b3d1-d8fac50e49c1 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:36.305954 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:36.348919 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeededc3-8447-40ad-a2c5-d1e4f57bdbdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:36.349560 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeededc3-8447-40ad-a2c5-d1e4f57bdbdf None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:36.349843 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeededc3-8447-40ad-a2c5-d1e4f57bdbdf None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:36.350117 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeededc3-8447-40ad-a2c5-d1e4f57bdbdf None admin] PATH_INFO: `/v3/groups/fb2a9e838fbd4936b9e7968ffe35e3eb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:36.351304 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeededc3-8447-40ad-a2c5-d1e4f57bdbdf None admin] RBAC: Authorizing `identity:get_group(group_id=fb2a9e838fbd4936b9e7968ffe35e3eb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:36.352802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeededc3-8447-40ad-a2c5-d1e4f57bdbdf None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:36.354695 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 215/417] 10.222.0.102 () {56 vars in 1185 bytes} [Wed Aug 24 15:33:36 2022] GET /identity/v3/groups/fb2a9e838fbd4936b9e7968ffe35e3eb => generated 230 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:36.355472 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeededc3-8447-40ad-a2c5-d1e4f57bdbdf None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:36.414789 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db37868-b100-4086-9a8f-f4ab67f3a6cf None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:36.460820 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d2ca8b6-43a9-4eb6-a34f-6e4cb85a8d09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:36.461463 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2ca8b6-43a9-4eb6-a34f-6e4cb85a8d09 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:36.461745 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2ca8b6-43a9-4eb6-a34f-6e4cb85a8d09 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:36.462086 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2ca8b6-43a9-4eb6-a34f-6e4cb85a8d09 None admin] PATH_INFO: `/v3/roles/member` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:36.476291 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeededc3-8447-40ad-a2c5-d1e4f57bdbdf None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:36.478761 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d2ca8b6-43a9-4eb6-a34f-6e4cb85a8d09 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:36.485460 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-4d2ca8b6-43a9-4eb6-a34f-6e4cb85a8d09 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:33:36.485460 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:36.485460 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:36.485460 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:36.485460 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:36.485460 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:36.485460 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:36.485460 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:36.485460 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:36.485460 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:36.485460 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:36.485460 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:36.485460 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:36.485460 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:36.485460 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:36.485460 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:36.486579 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:36.486579 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:36.486579 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:36.486579 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:36.486579 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:36.486579 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:36.486579 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:36.486579 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:36.486579 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:36.486579 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:36.486579 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:36.486579 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:36.486579 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:36.486579 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:36.486579 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:36.486579 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:36.487544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:36.487544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:36.487544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:36.487544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:36.487544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:36.487544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:36.487544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:36.487544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:36.487544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:36.487544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:36.487544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:33:36.487544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:36.489338 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 203/418] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:33:36 2022] GET /identity/v3/roles/member => generated 84 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:33:36.492867 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d2ca8b6-43a9-4eb6-a34f-6e4cb85a8d09 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:36.530719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca27d5a-0a41-4910-890a-47369deeda8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:36.531431 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca27d5a-0a41-4910-890a-47369deeda8e None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:36.531741 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca27d5a-0a41-4910-890a-47369deeda8e None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:36.532017 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca27d5a-0a41-4910-890a-47369deeda8e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:36.545700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca27d5a-0a41-4910-890a-47369deeda8e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:36.551571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-bca27d5a-0a41-4910-890a-47369deeda8e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 15:33:36.551571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:36.551571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:36.551571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:36.551571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:36.551571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:36.551571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:36.551571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:36.551571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:36.551571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:36.551571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:36.551571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:36.551571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:36.551571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:36.551571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:36.551571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:36.553466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:36.553466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:36.553466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:36.553466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:36.553466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:36.553466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:36.553466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:36.553466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:36.553466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:36.553466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:36.553466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:36.553466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:36.553466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:36.553466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:36.553466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:36.553466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:36.554896 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:36.554896 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:36.554896 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:36.554896 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:36.554896 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:36.554896 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:36.554896 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:36.554896 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:36.554896 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:36.554896 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:36.554896 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 15:33:36.554896 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:36.554896 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 216/419] 10.222.0.102 () {56 vars in 1119 bytes} [Wed Aug 24 15:33:36 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:36.554896 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca27d5a-0a41-4910-890a-47369deeda8e None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:36.573544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:36.600818 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:36.619821 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d2ca8b6-43a9-4eb6-a34f-6e4cb85a8d09 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:36.666864 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e34fb6-f5bd-4ce3-89a9-4f1b20569775 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:36.667519 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e34fb6-f5bd-4ce3-89a9-4f1b20569775 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:36.667984 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e34fb6-f5bd-4ce3-89a9-4f1b20569775 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:36.668359 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e34fb6-f5bd-4ce3-89a9-4f1b20569775 None admin] PATH_INFO: `/v3/projects/501a315437844349bdc02dafe337b855` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:36.669604 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e34fb6-f5bd-4ce3-89a9-4f1b20569775 None admin] RBAC: Authorizing `identity:get_project(project_id=501a315437844349bdc02dafe337b855)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:36.671871 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e34fb6-f5bd-4ce3-89a9-4f1b20569775 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:36.673681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 204/420] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:36 2022] GET /identity/v3/projects/501a315437844349bdc02dafe337b855 => generated 345 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:36.674534 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e34fb6-f5bd-4ce3-89a9-4f1b20569775 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:36.685374 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca27d5a-0a41-4910-890a-47369deeda8e None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:36.725705 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08c619d-e630-455a-86df-252d534462f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:36.726481 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08c619d-e630-455a-86df-252d534462f8 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:36.726795 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08c619d-e630-455a-86df-252d534462f8 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:36.727112 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08c619d-e630-455a-86df-252d534462f8 None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:36.727848 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08c619d-e630-455a-86df-252d534462f8 None admin] RBAC: Adding query filter params (name=member) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:36.728160 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08c619d-e630-455a-86df-252d534462f8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:36.729660 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08c619d-e630-455a-86df-252d534462f8 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:36.739355 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 217/421] 10.222.0.102 () {56 vars in 1106 bytes} [Wed Aug 24 15:33:36 2022] GET /identity/v3/roles?name=member => generated 343 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:36.740312 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08c619d-e630-455a-86df-252d534462f8 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:36.788757 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e34fb6-f5bd-4ce3-89a9-4f1b20569775 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:36.832150 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261db06c-6399-4570-9ccb-69bddf0862ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:36.832775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261db06c-6399-4570-9ccb-69bddf0862ad None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:36.833058 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261db06c-6399-4570-9ccb-69bddf0862ad None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:36.833866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261db06c-6399-4570-9ccb-69bddf0862ad None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:36.834562 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261db06c-6399-4570-9ccb-69bddf0862ad None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:36.835324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261db06c-6399-4570-9ccb-69bddf0862ad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:36.837209 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261db06c-6399-4570-9ccb-69bddf0862ad None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:36.840904 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:36.855727 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 205/422] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:33:36 2022] GET /identity/v3/roles?name=anotherrole => generated 336 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:36.856833 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261db06c-6399-4570-9ccb-69bddf0862ad None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:36.866553 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:36.875369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08c619d-e630-455a-86df-252d534462f8 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:36.920345 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc6db0a-77c0-4f54-9711-8a89147e29cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:36.921286 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc6db0a-77c0-4f54-9711-8a89147e29cf None admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:36.921712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc6db0a-77c0-4f54-9711-8a89147e29cf None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:36.922375 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc6db0a-77c0-4f54-9711-8a89147e29cf None admin] PATH_INFO: `/v3/projects/501a315437844349bdc02dafe337b855/groups/fb2a9e838fbd4936b9e7968ffe35e3eb/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:36.925390 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc6db0a-77c0-4f54-9711-8a89147e29cf None admin] RBAC: Authorizing `identity:create_grant(project_id=501a315437844349bdc02dafe337b855, group_id=fb2a9e838fbd4936b9e7968ffe35e3eb, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:36.927726 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc6db0a-77c0-4f54-9711-8a89147e29cf None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:36.953963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 218/423] 10.222.0.102 () {58 vars in 1447 bytes} [Wed Aug 24 15:33:36 2022] PUT /identity/v3/projects/501a315437844349bdc02dafe337b855/groups/fb2a9e838fbd4936b9e7968ffe35e3eb/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:33:36.955788 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc6db0a-77c0-4f54-9711-8a89147e29cf None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:37.028234 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261db06c-6399-4570-9ccb-69bddf0862ad None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:37.105791 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc6db0a-77c0-4f54-9711-8a89147e29cf None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:37.127798 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de4d06c-577d-4f83-a18f-699cb7adf3cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:37.128489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de4d06c-577d-4f83-a18f-699cb7adf3cd None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:37.128774 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de4d06c-577d-4f83-a18f-699cb7adf3cd None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:37.129047 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de4d06c-577d-4f83-a18f-699cb7adf3cd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:37.157743 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c8398c-4a61-43a6-a027-d5d7b32f9a45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:37.159387 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c8398c-4a61-43a6-a027-d5d7b32f9a45 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:37.159903 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c8398c-4a61-43a6-a027-d5d7b32f9a45 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:37.160326 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c8398c-4a61-43a6-a027-d5d7b32f9a45 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:37.196616 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:37.216413 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:37.223144 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de4d06c-577d-4f83-a18f-699cb7adf3cd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:37.224757 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de4d06c-577d-4f83-a18f-699cb7adf3cd None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:37.255505 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c8398c-4a61-43a6-a027-d5d7b32f9a45 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:37.257685 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c8398c-4a61-43a6-a027-d5d7b32f9a45 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:37.380078 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 206/424] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:33:36 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 523 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:33:37.385725 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de4d06c-577d-4f83-a18f-699cb7adf3cd None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:37.456091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 219/425] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:33:36 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 501 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:33:37.457279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3b75cb-b6b2-4306-acf6-b544d8d10c17 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:37.457564 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3b75cb-b6b2-4306-acf6-b544d8d10c17 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:37.457837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3b75cb-b6b2-4306-acf6-b544d8d10c17 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:37.459124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 220/426] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:37 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:37.465220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec3b75cb-b6b2-4306-acf6-b544d8d10c17 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:37.513166 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:37.520112 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de4d06c-577d-4f83-a18f-699cb7adf3cd None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:37.529116 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:37.561682 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dba2caf-b558-429a-a6a6-6f356a582d06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:37.562670 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dba2caf-b558-429a-a6a6-6f356a582d06 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:37.563019 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dba2caf-b558-429a-a6a6-6f356a582d06 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:37.563393 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dba2caf-b558-429a-a6a6-6f356a582d06 None admin] PATH_INFO: `/v3/projects/749c6d651fc445cdb75db25a73873386` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:37.564770 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dba2caf-b558-429a-a6a6-6f356a582d06 None admin] RBAC: Authorizing `identity:get_project(project_id=749c6d651fc445cdb75db25a73873386)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:37.566317 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dba2caf-b558-429a-a6a6-6f356a582d06 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:37.568394 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 207/427] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:37 2022] GET /identity/v3/projects/749c6d651fc445cdb75db25a73873386 => generated 299 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:33:37.569827 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bd5117-8b3e-48bb-b5fd-07da41dc672c None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:37.570113 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bd5117-8b3e-48bb-b5fd-07da41dc672c None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:37.570387 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bd5117-8b3e-48bb-b5fd-07da41dc672c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:37.576316 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec3b75cb-b6b2-4306-acf6-b544d8d10c17 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:37.589718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-08bd5117-8b3e-48bb-b5fd-07da41dc672c None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:37.591723 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-08bd5117-8b3e-48bb-b5fd-07da41dc672c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:37.613389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe00e81-64ba-4804-91eb-b7ef448be7ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:37.614193 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe00e81-64ba-4804-91eb-b7ef448be7ac None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:37.614578 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe00e81-64ba-4804-91eb-b7ef448be7ac None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:37.614944 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe00e81-64ba-4804-91eb-b7ef448be7ac None admin] PATH_INFO: `/v3/projects/749c6d651fc445cdb75db25a73873386` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:37.616252 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe00e81-64ba-4804-91eb-b7ef448be7ac None admin] RBAC: Authorizing `identity:get_project(project_id=749c6d651fc445cdb75db25a73873386)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:37.617837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe00e81-64ba-4804-91eb-b7ef448be7ac None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:37.624084 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 221/428] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:37 2022] GET /identity/v3/projects/749c6d651fc445cdb75db25a73873386 => generated 299 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:37.625088 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe00e81-64ba-4804-91eb-b7ef448be7ac None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:37.678633 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 208/429] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:37 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:37.679490 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08bd5117-8b3e-48bb-b5fd-07da41dc672c None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:37.787891 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:37.796289 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08bd5117-8b3e-48bb-b5fd-07da41dc672c None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:37.831658 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe00e81-64ba-4804-91eb-b7ef448be7ac None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:37.843226 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99b16fe-b005-421e-94f0-9851ace44281 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:37.846536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99b16fe-b005-421e-94f0-9851ace44281 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:37.846536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99b16fe-b005-421e-94f0-9851ace44281 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:37.846536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99b16fe-b005-421e-94f0-9851ace44281 None admin] PATH_INFO: `/v3/groups/6d47b38427704c8baaaefd6a3ae615a6` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:37.846536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99b16fe-b005-421e-94f0-9851ace44281 None admin] RBAC: Authorizing `identity:get_group(group_id=6d47b38427704c8baaaefd6a3ae615a6)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:37.847218 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99b16fe-b005-421e-94f0-9851ace44281 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:37.851665 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 209/430] 10.222.0.102 () {56 vars in 1185 bytes} [Wed Aug 24 15:33:37 2022] GET /identity/v3/groups/6d47b38427704c8baaaefd6a3ae615a6 => generated 227 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:37.856360 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99b16fe-b005-421e-94f0-9851ace44281 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:37.881422 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:37.907866 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d74eee5-e745-4416-9a46-1834848bef8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:37.908514 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d74eee5-e745-4416-9a46-1834848bef8f None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:37.908792 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d74eee5-e745-4416-9a46-1834848bef8f None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:37.909067 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d74eee5-e745-4416-9a46-1834848bef8f None admin] PATH_INFO: `/v3/groups/6d47b38427704c8baaaefd6a3ae615a6` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:37.910206 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d74eee5-e745-4416-9a46-1834848bef8f None admin] RBAC: Authorizing `identity:get_group(group_id=6d47b38427704c8baaaefd6a3ae615a6)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:37.911658 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d74eee5-e745-4416-9a46-1834848bef8f None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:37.913256 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 222/431] 10.222.0.102 () {56 vars in 1185 bytes} [Wed Aug 24 15:33:37 2022] GET /identity/v3/groups/6d47b38427704c8baaaefd6a3ae615a6 => generated 227 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:37.914426 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490005de-3696-4d5a-8ea1-d6116fab4b3e None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:37.914709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490005de-3696-4d5a-8ea1-d6116fab4b3e None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:37.914981 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490005de-3696-4d5a-8ea1-d6116fab4b3e None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:37.916608 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 223/432] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:37 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:37.920780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490005de-3696-4d5a-8ea1-d6116fab4b3e None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:38.007303 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99b16fe-b005-421e-94f0-9851ace44281 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:38.054167 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490005de-3696-4d5a-8ea1-d6116fab4b3e None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:38.060564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5565a987-2d0f-4181-8c30-950eb64a6720 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:38.061178 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5565a987-2d0f-4181-8c30-950eb64a6720 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:38.061457 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5565a987-2d0f-4181-8c30-950eb64a6720 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:38.061729 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5565a987-2d0f-4181-8c30-950eb64a6720 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:38.063176 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5565a987-2d0f-4181-8c30-950eb64a6720 None admin] RBAC: Adding query filter params (group.id=6d47b38427704c8baaaefd6a3ae615a6, role.id=b8429ebd9c2a4cbbbbf30ad56ce23bfa, scope.project.id=749c6d651fc445cdb75db25a73873386) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:38.063489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5565a987-2d0f-4181-8c30-950eb64a6720 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:38.065366 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5565a987-2d0f-4181-8c30-950eb64a6720 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:38.072225 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 210/433] 10.222.0.102 () {56 vars in 1381 bytes} [Wed Aug 24 15:33:37 2022] GET /identity/v3/role_assignments?group.id=6d47b38427704c8baaaefd6a3ae615a6&scope.project.id=749c6d651fc445cdb75db25a73873386&role.id=b8429ebd9c2a4cbbbbf30ad56ce23bfa => generated 613 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:38.080830 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aada167-3608-4c61-924c-16a7356eec82 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:38.081806 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aada167-3608-4c61-924c-16a7356eec82 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:38.082281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aada167-3608-4c61-924c-16a7356eec82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:38.096591 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c31fc9-1ade-47b0-8174-10aa02a2154d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:38.097220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c31fc9-1ade-47b0-8174-10aa02a2154d None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:38.097506 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c31fc9-1ade-47b0-8174-10aa02a2154d None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:38.097780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c31fc9-1ade-47b0-8174-10aa02a2154d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:38.098549 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c31fc9-1ade-47b0-8174-10aa02a2154d None admin] RBAC: Adding query filter params (group.id=6d47b38427704c8baaaefd6a3ae615a6, role.id=b5d2ed3e20ab47eeb741b4f43ab2eebb, scope.project.id=749c6d651fc445cdb75db25a73873386) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:38.098867 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c31fc9-1ade-47b0-8174-10aa02a2154d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:38.100482 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c31fc9-1ade-47b0-8174-10aa02a2154d None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:38.108233 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 224/434] 10.222.0.102 () {56 vars in 1381 bytes} [Wed Aug 24 15:33:37 2022] GET /identity/v3/role_assignments?group.id=6d47b38427704c8baaaefd6a3ae615a6&scope.project.id=749c6d651fc445cdb75db25a73873386&role.id=b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 613 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:38.109484 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c31fc9-1ade-47b0-8174-10aa02a2154d None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:38.111978 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-4aada167-3608-4c61-924c-16a7356eec82 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:38.115113 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-4aada167-3608-4c61-924c-16a7356eec82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:38.151789 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:38.219798 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 211/435] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:38 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:38.260558 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:38.339730 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c31fc9-1ade-47b0-8174-10aa02a2154d None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:38.378143 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db48b67-a570-415f-a527-22ccf5ef9fc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:38.378824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db48b67-a570-415f-a527-22ccf5ef9fc1 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:38.379283 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db48b67-a570-415f-a527-22ccf5ef9fc1 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:38.379562 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db48b67-a570-415f-a527-22ccf5ef9fc1 None admin] PATH_INFO: `/v3/roles/member` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:38.393214 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db48b67-a570-415f-a527-22ccf5ef9fc1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:38.398740 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-0db48b67-a570-415f-a527-22ccf5ef9fc1 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:33:38.398740 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:38.398740 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:38.398740 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:38.398740 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:38.398740 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:38.398740 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:38.398740 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:38.398740 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:38.398740 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:38.398740 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:38.398740 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:38.398740 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:38.398740 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:38.398740 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:38.398740 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:38.399676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:38.399676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:38.399676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:38.399676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:38.399676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:38.399676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:38.399676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:38.399676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:38.399676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:38.399676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:38.399676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:38.399676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:38.399676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:38.399676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:38.399676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:38.399676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:38.400665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:38.400665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:38.400665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:38.400665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:38.400665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:38.400665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:38.400665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:38.400665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:38.400665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:38.400665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:38.400665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 15:33:38.400665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:38.400665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 225/436] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:33:38 2022] GET /identity/v3/roles/member => generated 84 bytes in 291 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:38.406135 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aada167-3608-4c61-924c-16a7356eec82 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:38.519400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db48b67-a570-415f-a527-22ccf5ef9fc1 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:38.519921 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aada167-3608-4c61-924c-16a7356eec82 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:38.560933 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863a42fc-14f2-4f65-a645-14d98f942442 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:38.561559 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863a42fc-14f2-4f65-a645-14d98f942442 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:38.562091 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863a42fc-14f2-4f65-a645-14d98f942442 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:38.562363 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863a42fc-14f2-4f65-a645-14d98f942442 None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:38.563517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-863a42fc-14f2-4f65-a645-14d98f942442 None admin] RBAC: Adding query filter params (name=member) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:38.563517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-863a42fc-14f2-4f65-a645-14d98f942442 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:38.564727 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-863a42fc-14f2-4f65-a645-14d98f942442 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:38.574086 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 212/437] 10.222.0.102 () {56 vars in 1106 bytes} [Wed Aug 24 15:33:38 2022] GET /identity/v3/roles?name=member => generated 343 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:38.580153 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863a42fc-14f2-4f65-a645-14d98f942442 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:38.610876 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:38.671156 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db48b67-a570-415f-a527-22ccf5ef9fc1 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:38.699400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:38.718875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863a42fc-14f2-4f65-a645-14d98f942442 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:38.725132 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419630e0-9d8d-45b7-bd27-360432360406 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:38.725782 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419630e0-9d8d-45b7-bd27-360432360406 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:38.726067 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419630e0-9d8d-45b7-bd27-360432360406 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:38.726364 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419630e0-9d8d-45b7-bd27-360432360406 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:38.743272 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-419630e0-9d8d-45b7-bd27-360432360406 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:38.748670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-419630e0-9d8d-45b7-bd27-360432360406 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 15:33:38.748670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:38.748670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:38.748670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:38.748670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:38.748670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:38.748670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:38.748670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:38.748670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:38.748670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:38.748670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:38.748670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:38.748670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:38.748670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:38.748670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:38.748670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:38.749590 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:38.749590 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:38.749590 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:38.749590 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:38.749590 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:38.749590 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:38.749590 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:38.749590 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:38.749590 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:38.749590 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:38.749590 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:38.749590 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:38.749590 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:38.749590 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:38.749590 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:38.749590 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:38.750774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:38.750774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:38.750774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:38.750774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:38.750774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:38.750774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:38.750774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:38.750774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:38.750774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:38.750774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:38.750774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 15:33:38.750774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:38.750774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 226/438] 10.222.0.102 () {56 vars in 1119 bytes} [Wed Aug 24 15:33:38 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:33:38.759067 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419630e0-9d8d-45b7-bd27-360432360406 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:38.770312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b901ba-bd18-47ad-94c8-363e979a6ce6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:38.770663 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b901ba-bd18-47ad-94c8-363e979a6ce6 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:38.770949 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b901ba-bd18-47ad-94c8-363e979a6ce6 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:38.771225 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b901ba-bd18-47ad-94c8-363e979a6ce6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:38.892700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b901ba-bd18-47ad-94c8-363e979a6ce6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:38.894713 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b901ba-bd18-47ad-94c8-363e979a6ce6 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:38.916011 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419630e0-9d8d-45b7-bd27-360432360406 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:38.928996 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:38.972259 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f096b5b-c5fb-4fad-81f7-fc32884ca3c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:38.972741 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f096b5b-c5fb-4fad-81f7-fc32884ca3c3 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:38.972869 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f096b5b-c5fb-4fad-81f7-fc32884ca3c3 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:38.973072 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f096b5b-c5fb-4fad-81f7-fc32884ca3c3 None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:38.973755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f096b5b-c5fb-4fad-81f7-fc32884ca3c3 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:38.974063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f096b5b-c5fb-4fad-81f7-fc32884ca3c3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:38.975811 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f096b5b-c5fb-4fad-81f7-fc32884ca3c3 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:38.986800 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 227/439] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:33:38 2022] GET /identity/v3/roles?name=anotherrole => generated 336 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:33:38.991989 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f096b5b-c5fb-4fad-81f7-fc32884ca3c3 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:39.029073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 213/440] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:33:38 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 450 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 24 15:33:39.034632 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b901ba-bd18-47ad-94c8-363e979a6ce6 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:39.041481 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:39.105804 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f096b5b-c5fb-4fad-81f7-fc32884ca3c3 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:39.141895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c00ae3-3fb1-4603-ae97-bdca63aa4ada None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:39.142601 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c00ae3-3fb1-4603-ae97-bdca63aa4ada None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:39.142903 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c00ae3-3fb1-4603-ae97-bdca63aa4ada None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:39.143090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c00ae3-3fb1-4603-ae97-bdca63aa4ada None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:39.155065 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b901ba-bd18-47ad-94c8-363e979a6ce6 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:39.179939 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:39.198015 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94fbe50-95c4-4e7b-93ce-545219c711d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:39.198762 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94fbe50-95c4-4e7b-93ce-545219c711d9 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:39.199125 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94fbe50-95c4-4e7b-93ce-545219c711d9 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:39.199411 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94fbe50-95c4-4e7b-93ce-545219c711d9 None admin] PATH_INFO: `/v3/projects/86a6b464bc2d4f72895f92c759e26a36` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:39.200608 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94fbe50-95c4-4e7b-93ce-545219c711d9 None admin] RBAC: Authorizing `identity:get_project(project_id=86a6b464bc2d4f72895f92c759e26a36)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:39.202755 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94fbe50-95c4-4e7b-93ce-545219c711d9 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:39.204469 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 214/441] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:39 2022] GET /identity/v3/projects/86a6b464bc2d4f72895f92c759e26a36 => generated 303 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:33:39.209845 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94fbe50-95c4-4e7b-93ce-545219c711d9 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:39.229625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c00ae3-3fb1-4603-ae97-bdca63aa4ada None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:39.231459 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c00ae3-3fb1-4603-ae97-bdca63aa4ada None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:39.275437 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:39.316620 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94fbe50-95c4-4e7b-93ce-545219c711d9 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:39.339954 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 228/442] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:33:38 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:33:39.341341 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b10a311-b6a6-47a3-961d-d15120933aff None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:39.341623 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b10a311-b6a6-47a3-961d-d15120933aff None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:39.341949 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b10a311-b6a6-47a3-961d-d15120933aff None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:39.343662 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 229/443] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:39 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:39.345915 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b10a311-b6a6-47a3-961d-d15120933aff None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:39.356389 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45048abd-081a-4d9a-9640-8d71b480031d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:39.357011 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45048abd-081a-4d9a-9640-8d71b480031d None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:39.357286 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45048abd-081a-4d9a-9640-8d71b480031d None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:39.358091 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45048abd-081a-4d9a-9640-8d71b480031d None admin] PATH_INFO: `/v3/groups/6d47b38427704c8baaaefd6a3ae615a6` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:39.359212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45048abd-081a-4d9a-9640-8d71b480031d None admin] RBAC: Authorizing `identity:get_group(group_id=6d47b38427704c8baaaefd6a3ae615a6)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:39.360689 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45048abd-081a-4d9a-9640-8d71b480031d None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:39.362252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 215/444] 10.222.0.102 () {56 vars in 1185 bytes} [Wed Aug 24 15:33:39 2022] GET /identity/v3/groups/6d47b38427704c8baaaefd6a3ae615a6 => generated 227 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:33:39.362252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef8cb9f-7241-471b-b0c1-5ec249beabac None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:39.362252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef8cb9f-7241-471b-b0c1-5ec249beabac None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:39.362252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef8cb9f-7241-471b-b0c1-5ec249beabac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:39.382483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-eef8cb9f-7241-471b-b0c1-5ec249beabac None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:39.383712 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-eef8cb9f-7241-471b-b0c1-5ec249beabac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:39.439548 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:39.451538 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b10a311-b6a6-47a3-961d-d15120933aff None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:39.458444 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 216/445] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:39 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:39.459748 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef8cb9f-7241-471b-b0c1-5ec249beabac None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:39.487085 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78aa05d-952a-40f3-bc7c-c420f68c1173 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:39.487792 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78aa05d-952a-40f3-bc7c-c420f68c1173 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:39.488108 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78aa05d-952a-40f3-bc7c-c420f68c1173 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:39.488395 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78aa05d-952a-40f3-bc7c-c420f68c1173 None admin] PATH_INFO: `/v3/projects/86a6b464bc2d4f72895f92c759e26a36` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:39.489515 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b78aa05d-952a-40f3-bc7c-c420f68c1173 None admin] RBAC: Authorizing `identity:get_project(project_id=86a6b464bc2d4f72895f92c759e26a36)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:39.490968 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b78aa05d-952a-40f3-bc7c-c420f68c1173 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:39.492619 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 230/446] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:39 2022] GET /identity/v3/projects/86a6b464bc2d4f72895f92c759e26a36 => generated 303 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:33:39.499405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78aa05d-952a-40f3-bc7c-c420f68c1173 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:39.509566 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:39.572591 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef8cb9f-7241-471b-b0c1-5ec249beabac None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:39.610102 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038ccdca-964c-41ac-90dc-baa0ab2cb0c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:39.610706 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038ccdca-964c-41ac-90dc-baa0ab2cb0c9 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:39.610986 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038ccdca-964c-41ac-90dc-baa0ab2cb0c9 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:39.611509 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78aa05d-952a-40f3-bc7c-c420f68c1173 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:39.611585 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038ccdca-964c-41ac-90dc-baa0ab2cb0c9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:39.612013 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038ccdca-964c-41ac-90dc-baa0ab2cb0c9 None admin] RBAC: Adding query filter params (group.id=6d47b38427704c8baaaefd6a3ae615a6, role.id=b5d2ed3e20ab47eeb741b4f43ab2eebb, scope.project.id=86a6b464bc2d4f72895f92c759e26a36) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:39.612325 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038ccdca-964c-41ac-90dc-baa0ab2cb0c9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:39.613763 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038ccdca-964c-41ac-90dc-baa0ab2cb0c9 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:39.620218 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 217/447] 10.222.0.102 () {56 vars in 1381 bytes} [Wed Aug 24 15:33:39 2022] GET /identity/v3/role_assignments?group.id=6d47b38427704c8baaaefd6a3ae615a6&scope.project.id=86a6b464bc2d4f72895f92c759e26a36&role.id=b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 613 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:39.650076 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f726df2a-486d-4b5c-943a-7981d22abb90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:39.650734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f726df2a-486d-4b5c-943a-7981d22abb90 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:39.651017 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f726df2a-486d-4b5c-943a-7981d22abb90 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:39.651293 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f726df2a-486d-4b5c-943a-7981d22abb90 None admin] PATH_INFO: `/v3/groups/6d47b38427704c8baaaefd6a3ae615a6` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:39.652594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f726df2a-486d-4b5c-943a-7981d22abb90 None admin] RBAC: Authorizing `identity:get_group(group_id=6d47b38427704c8baaaefd6a3ae615a6)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:39.654028 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f726df2a-486d-4b5c-943a-7981d22abb90 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:39.656832 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 231/448] 10.222.0.102 () {56 vars in 1185 bytes} [Wed Aug 24 15:33:39 2022] GET /identity/v3/groups/6d47b38427704c8baaaefd6a3ae615a6 => generated 227 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:33:39.662599 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038ccdca-964c-41ac-90dc-baa0ab2cb0c9 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:39.728661 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:39.730173 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f726df2a-486d-4b5c-943a-7981d22abb90 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:39.796899 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038ccdca-964c-41ac-90dc-baa0ab2cb0c9 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:39.830654 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:39.841127 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4f5ce9-45ca-4e2b-a1c2-38ebcf40757f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:39.841772 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4f5ce9-45ca-4e2b-a1c2-38ebcf40757f None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:39.842555 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4f5ce9-45ca-4e2b-a1c2-38ebcf40757f None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:39.843409 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4f5ce9-45ca-4e2b-a1c2-38ebcf40757f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:39.844613 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4f5ce9-45ca-4e2b-a1c2-38ebcf40757f None admin] RBAC: Adding query filter params (group.id=6d47b38427704c8baaaefd6a3ae615a6, role.id=b8429ebd9c2a4cbbbbf30ad56ce23bfa, scope.project.id=86a6b464bc2d4f72895f92c759e26a36) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:39.844613 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4f5ce9-45ca-4e2b-a1c2-38ebcf40757f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:39.845939 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4f5ce9-45ca-4e2b-a1c2-38ebcf40757f None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:39.855195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 218/449] 10.222.0.102 () {56 vars in 1381 bytes} [Wed Aug 24 15:33:39 2022] GET /identity/v3/role_assignments?group.id=6d47b38427704c8baaaefd6a3ae615a6&scope.project.id=86a6b464bc2d4f72895f92c759e26a36&role.id=b8429ebd9c2a4cbbbbf30ad56ce23bfa => generated 613 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:39.867175 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f726df2a-486d-4b5c-943a-7981d22abb90 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:39.914185 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9131b040-702b-4e30-adb2-2da5228cca94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:39.914845 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9131b040-702b-4e30-adb2-2da5228cca94 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:39.915129 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9131b040-702b-4e30-adb2-2da5228cca94 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:39.915401 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9131b040-702b-4e30-adb2-2da5228cca94 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:39.929155 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9131b040-702b-4e30-adb2-2da5228cca94 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:39.934288 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-9131b040-702b-4e30-adb2-2da5228cca94 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:33:39.934288 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:39.934288 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:39.934288 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:39.934288 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:39.934288 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:39.934288 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:39.934288 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:39.934288 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:39.934288 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:39.934288 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:39.934288 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:39.934288 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:39.934288 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:39.934288 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:39.934288 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:39.935293 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:39.935293 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:39.935293 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:39.935293 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:39.935293 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:39.935293 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:39.935293 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:39.935293 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:39.935293 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:39.935293 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:39.935293 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:39.935293 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:39.935293 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:39.935293 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:39.935293 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:39.935293 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:39.937108 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:39.937108 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:39.937108 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:39.937108 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:39.937108 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:39.937108 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:39.937108 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:39.937108 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:39.937108 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:39.937108 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:39.937108 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:33:39.937108 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:39.937108 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 232/450] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:33:39 2022] GET /identity/v3/roles/admin => generated 83 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:33:39.943503 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4f5ce9-45ca-4e2b-a1c2-38ebcf40757f None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:40.065121 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4f5ce9-45ca-4e2b-a1c2-38ebcf40757f None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:40.082064 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:40.103376 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c0f450-b642-4be1-8ae5-411f0ba6785b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:40.104012 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c0f450-b642-4be1-8ae5-411f0ba6785b None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:40.104284 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c0f450-b642-4be1-8ae5-411f0ba6785b None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:40.104603 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c0f450-b642-4be1-8ae5-411f0ba6785b None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:40.105223 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39c0f450-b642-4be1-8ae5-411f0ba6785b None admin] RBAC: Adding query filter params (name=admin) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:40.105524 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39c0f450-b642-4be1-8ae5-411f0ba6785b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:40.106899 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39c0f450-b642-4be1-8ae5-411f0ba6785b None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:40.117111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 219/451] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:33:39 2022] GET /identity/v3/roles?name=admin => generated 341 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:40.122929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9131b040-702b-4e30-adb2-2da5228cca94 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:40.230322 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9131b040-702b-4e30-adb2-2da5228cca94 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:40.265485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb082a4e-0c8b-46ab-bdd4-f4f225569b0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:40.266113 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb082a4e-0c8b-46ab-bdd4-f4f225569b0a None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:40.266380 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb082a4e-0c8b-46ab-bdd4-f4f225569b0a None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:40.266656 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb082a4e-0c8b-46ab-bdd4-f4f225569b0a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:40.269460 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:40.354018 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb082a4e-0c8b-46ab-bdd4-f4f225569b0a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:40.355932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb082a4e-0c8b-46ab-bdd4-f4f225569b0a None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:40.460738 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 233/452] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:33:40 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:33:40.466621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c0f450-b642-4be1-8ae5-411f0ba6785b None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:40.564400 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c0f450-b642-4be1-8ae5-411f0ba6785b None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:40.597405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19383ba-5d89-4ab9-9e87-9284afd67725 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:40.598015 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19383ba-5d89-4ab9-9e87-9284afd67725 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:40.598295 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19383ba-5d89-4ab9-9e87-9284afd67725 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:40.598567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19383ba-5d89-4ab9-9e87-9284afd67725 None admin] PATH_INFO: `/v3/projects/501a315437844349bdc02dafe337b855` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:40.599706 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19383ba-5d89-4ab9-9e87-9284afd67725 None admin] RBAC: Authorizing `identity:get_project(project_id=501a315437844349bdc02dafe337b855)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:40.601139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19383ba-5d89-4ab9-9e87-9284afd67725 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:40.602778 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 220/453] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:33:40 2022] GET /identity/v3/projects/501a315437844349bdc02dafe337b855 => generated 345 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:40.609268 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb082a4e-0c8b-46ab-bdd4-f4f225569b0a None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:40.641969 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:40.718994 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb082a4e-0c8b-46ab-bdd4-f4f225569b0a None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:40.753956 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25c5171-9c90-400f-98da-dbd619231e14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:40.754658 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25c5171-9c90-400f-98da-dbd619231e14 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:40.754846 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25c5171-9c90-400f-98da-dbd619231e14 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:40.755112 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25c5171-9c90-400f-98da-dbd619231e14 None admin] PATH_INFO: `/v3/groups/fb2a9e838fbd4936b9e7968ffe35e3eb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:40.756221 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25c5171-9c90-400f-98da-dbd619231e14 None admin] RBAC: Authorizing `identity:get_group(group_id=fb2a9e838fbd4936b9e7968ffe35e3eb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:40.757652 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25c5171-9c90-400f-98da-dbd619231e14 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:40.759199 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 234/454] 10.222.0.102 () {56 vars in 1185 bytes} [Wed Aug 24 15:33:40 2022] GET /identity/v3/groups/fb2a9e838fbd4936b9e7968ffe35e3eb => generated 230 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:33:40.765639 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19383ba-5d89-4ab9-9e87-9284afd67725 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:40.813870 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:40.872536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19383ba-5d89-4ab9-9e87-9284afd67725 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:40.906231 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d5cec02-9b3a-4d9c-b05e-d9dceb2dcb9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:40.906812 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5cec02-9b3a-4d9c-b05e-d9dceb2dcb9a None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:40.907089 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5cec02-9b3a-4d9c-b05e-d9dceb2dcb9a None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:40.907357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5cec02-9b3a-4d9c-b05e-d9dceb2dcb9a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:40.908112 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d5cec02-9b3a-4d9c-b05e-d9dceb2dcb9a None admin] RBAC: Adding query filter params (group.id=fb2a9e838fbd4936b9e7968ffe35e3eb, role.id=4d8fbbaea81a4145b89af12350bb067c, scope.project.id=501a315437844349bdc02dafe337b855) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:40.908419 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d5cec02-9b3a-4d9c-b05e-d9dceb2dcb9a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:40.909833 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d5cec02-9b3a-4d9c-b05e-d9dceb2dcb9a None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:40.915495 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 221/455] 10.222.0.102 () {56 vars in 1381 bytes} [Wed Aug 24 15:33:40 2022] GET /identity/v3/role_assignments?group.id=fb2a9e838fbd4936b9e7968ffe35e3eb&scope.project.id=501a315437844349bdc02dafe337b855&role.id=4d8fbbaea81a4145b89af12350bb067c => generated 613 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:43.285374 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7caa60f5-d902-4130-a378-1798ebadb586 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:43.286417 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7caa60f5-d902-4130-a378-1798ebadb586 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:43.286894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7caa60f5-d902-4130-a378-1798ebadb586 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:43.288721 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 235/456] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:43 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:43.294335 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24087ef3-f0d6-4748-978b-09b89c1029e1 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:43.295778 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24087ef3-f0d6-4748-978b-09b89c1029e1 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:43.296294 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24087ef3-f0d6-4748-978b-09b89c1029e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:43.319804 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-24087ef3-f0d6-4748-978b-09b89c1029e1 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:43.321341 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-24087ef3-f0d6-4748-978b-09b89c1029e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:43.414926 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 222/457] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:43 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:43.476895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87085dd6-1e75-44d9-af47-6f1bcd132705 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:43.477175 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87085dd6-1e75-44d9-af47-6f1bcd132705 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:43.477620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87085dd6-1e75-44d9-af47-6f1bcd132705 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:43.479114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 236/458] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:43 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:43.484751 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40fa4458-2b7e-4a8e-b08a-e119c12b830b None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:43.485296 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40fa4458-2b7e-4a8e-b08a-e119c12b830b None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:43.485822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40fa4458-2b7e-4a8e-b08a-e119c12b830b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:43.505288 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-40fa4458-2b7e-4a8e-b08a-e119c12b830b None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:43.506871 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-40fa4458-2b7e-4a8e-b08a-e119c12b830b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:43.539245 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:43.597777 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 223/459] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:43 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:43.786731 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87085dd6-1e75-44d9-af47-6f1bcd132705 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:43.814994 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448d745b-1ab3-4efa-9da6-815fdb334772 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:43.815622 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448d745b-1ab3-4efa-9da6-815fdb334772 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:43.816121 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448d745b-1ab3-4efa-9da6-815fdb334772 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:43.817941 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 224/460] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:43 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:43.829113 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77bd886-10c4-43a6-b0b0-156fcc880daa None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:43.829737 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77bd886-10c4-43a6-b0b0-156fcc880daa None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:43.830226 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77bd886-10c4-43a6-b0b0-156fcc880daa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:43.859228 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-d77bd886-10c4-43a6-b0b0-156fcc880daa None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:43.862261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-d77bd886-10c4-43a6-b0b0-156fcc880daa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:43.936261 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87085dd6-1e75-44d9-af47-6f1bcd132705 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:43.939582 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:43.980188 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48a56ab-39f8-41d8-8a65-c3e583076099 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:43.981248 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48a56ab-39f8-41d8-8a65-c3e583076099 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:43.981700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48a56ab-39f8-41d8-8a65-c3e583076099 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:43.982114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48a56ab-39f8-41d8-8a65-c3e583076099 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:43.991904 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 225/461] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:43 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:43.993167 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77bd886-10c4-43a6-b0b0-156fcc880daa None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:43.999069 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48a56ab-39f8-41d8-8a65-c3e583076099 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:44.000746 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48a56ab-39f8-41d8-8a65-c3e583076099 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:44.026761 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-a48a56ab-39f8-41d8-8a65-c3e583076099 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:44.026761 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:44.026761 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:44.026761 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:44.026761 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:44.026761 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:44.026761 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:44.026761 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:44.026761 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:44.026761 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:44.026761 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:44.026761 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:44.026761 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:33:44.026761 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:33:44.026761 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:33:44.026761 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:33:44.027966 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:44.027966 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:44.027966 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:44.027966 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:44.027966 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:44.027966 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:44.027966 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:44.027966 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:44.027966 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:44.027966 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:44.027966 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:44.027966 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:44.027966 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:44.027966 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:44.027966 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:44.027966 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:44.027966 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:33:44.029020 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:33:44.029020 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:44.029020 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:44.037312 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 237/462] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:33:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:44.043065 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48a56ab-39f8-41d8-8a65-c3e583076099 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:44.187571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:44.195521 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77bd886-10c4-43a6-b0b0-156fcc880daa None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:44.249006 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48a56ab-39f8-41d8-8a65-c3e583076099 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:44.262576 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36069e8d-87ef-4bb7-bfeb-5b0a89f59ddc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:44.263718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36069e8d-87ef-4bb7-bfeb-5b0a89f59ddc None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:44.264207 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36069e8d-87ef-4bb7-bfeb-5b0a89f59ddc None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:44.264720 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36069e8d-87ef-4bb7-bfeb-5b0a89f59ddc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:44.277658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36069e8d-87ef-4bb7-bfeb-5b0a89f59ddc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:44.280268 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36069e8d-87ef-4bb7-bfeb-5b0a89f59ddc None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:44.298529 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-36069e8d-87ef-4bb7-bfeb-5b0a89f59ddc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:44.298529 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:44.298529 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:44.298529 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:44.298529 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:44.298529 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:44.298529 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:44.298529 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:44.298529 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:44.298529 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:44.298529 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:44.298529 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:44.298529 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:33:44.298529 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:33:44.298529 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:33:44.298529 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:33:44.299963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:44.299963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:44.299963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:44.299963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:44.299963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:44.299963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:44.299963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:44.299963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:44.299963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:44.299963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:44.299963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:44.299963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:44.299963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:44.299963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:44.299963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:44.299963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:44.299963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:33:44.300792 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:33:44.300792 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:44.300792 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:44.303306 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 226/463] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:33:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 310 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:44.309934 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36069e8d-87ef-4bb7-bfeb-5b0a89f59ddc None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:44.339997 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fd3798-cc8e-4102-817b-b69854be7979 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:44.340665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fd3798-cc8e-4102-817b-b69854be7979 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:44.340953 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fd3798-cc8e-4102-817b-b69854be7979 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:44.341227 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fd3798-cc8e-4102-817b-b69854be7979 None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:44.342430 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1fd3798-cc8e-4102-817b-b69854be7979 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:44.342906 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1fd3798-cc8e-4102-817b-b69854be7979 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:44.344346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1fd3798-cc8e-4102-817b-b69854be7979 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:44.380043 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 238/464] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:33:44 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 337 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:44.381795 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fd3798-cc8e-4102-817b-b69854be7979 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:44.400222 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:44.453891 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36069e8d-87ef-4bb7-bfeb-5b0a89f59ddc None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:44.492562 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1dd379-2da4-4817-80e6-9e0cce20a4cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:44.493352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1dd379-2da4-4817-80e6-9e0cce20a4cd None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:44.493637 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1dd379-2da4-4817-80e6-9e0cce20a4cd None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:44.493910 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1dd379-2da4-4817-80e6-9e0cce20a4cd None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:44.494572 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1dd379-2da4-4817-80e6-9e0cce20a4cd None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:44.494879 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1dd379-2da4-4817-80e6-9e0cce20a4cd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:44.496287 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1dd379-2da4-4817-80e6-9e0cce20a4cd None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:44.496636 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fd3798-cc8e-4102-817b-b69854be7979 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:44.510152 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 227/465] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:33:44 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:33:44.510803 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1dd379-2da4-4817-80e6-9e0cce20a4cd None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:44.525105 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:44.540316 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b418ec9d-0090-4a3f-96f6-831ffdba1794 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:44.540956 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b418ec9d-0090-4a3f-96f6-831ffdba1794 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:44.541355 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b418ec9d-0090-4a3f-96f6-831ffdba1794 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:44.541632 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b418ec9d-0090-4a3f-96f6-831ffdba1794 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:44.554113 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b418ec9d-0090-4a3f-96f6-831ffdba1794 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:44.555873 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b418ec9d-0090-4a3f-96f6-831ffdba1794 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:44.574572 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-b418ec9d-0090-4a3f-96f6-831ffdba1794 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:44.574572 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:44.574572 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:44.574572 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:44.574572 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:44.574572 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:44.574572 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:44.574572 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:44.574572 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:44.574572 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:44.574572 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:44.574572 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:44.574572 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:33:44.574572 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:33:44.574572 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:33:44.574572 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:33:44.575516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:44.575516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:44.575516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:44.575516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:44.575516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:44.575516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:44.575516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:44.575516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:44.575516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:44.575516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:44.575516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:44.575516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:44.575516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:44.575516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:44.575516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:44.575516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:44.575516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:33:44.576510 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:33:44.576510 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:44.576510 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:44.576510 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 239/466] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:33:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:44.577346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb570da5-26d2-4fad-bb51-78556a60149f None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:44.577632 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb570da5-26d2-4fad-bb51-78556a60149f None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:44.577906 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb570da5-26d2-4fad-bb51-78556a60149f None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:44.583352 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 240/467] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:44 2022] GET /identity => generated 271 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:44.584174 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb570da5-26d2-4fad-bb51-78556a60149f None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:44.628195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1dd379-2da4-4817-80e6-9e0cce20a4cd None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:44.668314 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec55cf28-71dc-4150-8d95-0eb3dd33dc28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:44.668529 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:44.668964 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec55cf28-71dc-4150-8d95-0eb3dd33dc28 None admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:44.669294 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec55cf28-71dc-4150-8d95-0eb3dd33dc28 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:44.669571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec55cf28-71dc-4150-8d95-0eb3dd33dc28 None admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:44.670871 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec55cf28-71dc-4150-8d95-0eb3dd33dc28 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:44.672244 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec55cf28-71dc-4150-8d95-0eb3dd33dc28 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:44.697072 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 228/468] 10.222.0.102 () {60 vars in 1137 bytes} [Wed Aug 24 15:33:44 2022] POST /identity/v3/users => generated 247 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:33:44.697906 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec55cf28-71dc-4150-8d95-0eb3dd33dc28 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:44.728207 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb570da5-26d2-4fad-bb51-78556a60149f None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:44.786295 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f708c539-7444-4416-abf8-7c8f911077cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:44.787671 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f708c539-7444-4416-abf8-7c8f911077cf None admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:44.788710 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f708c539-7444-4416-abf8-7c8f911077cf None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:44.789340 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f708c539-7444-4416-abf8-7c8f911077cf None admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:44.790402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f708c539-7444-4416-abf8-7c8f911077cf None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:44.792789 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:44.793010 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f708c539-7444-4416-abf8-7c8f911077cf None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:44.831588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 241/469] 10.222.0.102 () {60 vars in 1137 bytes} [Wed Aug 24 15:33:44 2022] POST /identity/v3/users => generated 249 bytes in 248 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:33:44.833212 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519a580d-a770-4fe7-ad42-e4dafdfa2c66 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:44.833623 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519a580d-a770-4fe7-ad42-e4dafdfa2c66 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:44.833975 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519a580d-a770-4fe7-ad42-e4dafdfa2c66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:44.851919 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec55cf28-71dc-4150-8d95-0eb3dd33dc28 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:44.856134 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-519a580d-a770-4fe7-ad42-e4dafdfa2c66 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:44.857546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-519a580d-a770-4fe7-ad42-e4dafdfa2c66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:44.908830 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c64f5f-e363-400e-85f7-b95446f894d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:44.909731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c64f5f-e363-400e-85f7-b95446f894d5 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:44.910151 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c64f5f-e363-400e-85f7-b95446f894d5 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:44.910572 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c64f5f-e363-400e-85f7-b95446f894d5 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:44.911473 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c64f5f-e363-400e-85f7-b95446f894d5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:44.912176 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c64f5f-e363-400e-85f7-b95446f894d5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:44.913975 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c64f5f-e363-400e-85f7-b95446f894d5 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:44.933113 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 229/470] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:33:44 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:44.940084 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c64f5f-e363-400e-85f7-b95446f894d5 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:44.962744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 242/471] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:44 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:44.964184 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626f985b-bd73-4406-a3ee-9045d3a9d4ff None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:44.964470 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626f985b-bd73-4406-a3ee-9045d3a9d4ff None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:44.964743 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626f985b-bd73-4406-a3ee-9045d3a9d4ff None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:44.966256 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 243/472] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:44 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:44.973369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d95ac6-a4c0-40b2-8b0e-772a28a7b534 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:44.973664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d95ac6-a4c0-40b2-8b0e-772a28a7b534 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:44.973945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d95ac6-a4c0-40b2-8b0e-772a28a7b534 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:44.996861 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-c9d95ac6-a4c0-40b2-8b0e-772a28a7b534 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:44.998587 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-c9d95ac6-a4c0-40b2-8b0e-772a28a7b534 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:45.070399 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:45.084553 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c64f5f-e363-400e-85f7-b95446f894d5 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:45.091030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 244/473] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:44 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:45.133868 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:45.150342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642838ab-f02e-491f-a178-daf06c920b37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:45.150968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642838ab-f02e-491f-a178-daf06c920b37 None admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:45.151248 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642838ab-f02e-491f-a178-daf06c920b37 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:45.151530 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642838ab-f02e-491f-a178-daf06c920b37 None admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:45.152250 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642838ab-f02e-491f-a178-daf06c920b37 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:45.153748 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642838ab-f02e-491f-a178-daf06c920b37 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:45.182543 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 230/474] 10.222.0.102 () {60 vars in 1137 bytes} [Wed Aug 24 15:33:44 2022] POST /identity/v3/users => generated 249 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 15:33:45.303245 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d95ac6-a4c0-40b2-8b0e-772a28a7b534 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:45.419426 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d95ac6-a4c0-40b2-8b0e-772a28a7b534 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:45.459673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593b1f91-6645-4d56-b539-8dd41e9db7ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:45.460316 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593b1f91-6645-4d56-b539-8dd41e9db7ae None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:45.460597 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593b1f91-6645-4d56-b539-8dd41e9db7ae None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:45.460931 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593b1f91-6645-4d56-b539-8dd41e9db7ae None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:45.469458 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642838ab-f02e-491f-a178-daf06c920b37 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:45.475514 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-593b1f91-6645-4d56-b539-8dd41e9db7ae None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:45.477281 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-593b1f91-6645-4d56-b539-8dd41e9db7ae None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:45.494153 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-593b1f91-6645-4d56-b539-8dd41e9db7ae None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:45.494153 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:45.494153 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:45.494153 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:45.494153 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:45.494153 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:45.494153 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:45.494153 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:45.494153 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:45.494153 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:45.494153 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:45.494153 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:45.494153 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:33:45.494153 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:33:45.494153 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:33:45.494153 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:33:45.495091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:45.495091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:45.495091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:45.495091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:45.495091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:45.495091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:45.495091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:45.495091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:45.495091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:45.495091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:45.495091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:45.495091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:45.495091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:45.495091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:45.495091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:45.495091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:45.495091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:33:45.495945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:33:45.495945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:45.495945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:45.497899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 245/475] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:33:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:45.509063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593b1f91-6645-4d56-b539-8dd41e9db7ae None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:45.558198 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:45.627329 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642838ab-f02e-491f-a178-daf06c920b37 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:45.655653 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593b1f91-6645-4d56-b539-8dd41e9db7ae None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:45.675708 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9615c563-fc7b-4b1a-8ae6-f64123c18638 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:45.676329 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9615c563-fc7b-4b1a-8ae6-f64123c18638 None admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:45.676607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9615c563-fc7b-4b1a-8ae6-f64123c18638 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:45.676879 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9615c563-fc7b-4b1a-8ae6-f64123c18638 None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:45.677565 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9615c563-fc7b-4b1a-8ae6-f64123c18638 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:45.679059 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9615c563-fc7b-4b1a-8ae6-f64123c18638 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:45.692682 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.sql.core [None req-9615c563-fc7b-4b1a-8ae6-f64123c18638 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 24 15:33:45.692682 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Aug 24 15:33:45.692682 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [parameters: {'id': 'd795512273704103b0fb77fe7c377c24', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Aug 24 15:33:45.692682 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=88218) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 24 15:33:45.693729 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb58931c-9eb3-4363-b340-5aa9e64ed348 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:45.694398 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb58931c-9eb3-4363-b340-5aa9e64ed348 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:45.694684 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb58931c-9eb3-4363-b340-5aa9e64ed348 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:45.694955 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb58931c-9eb3-4363-b340-5aa9e64ed348 None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:45.695605 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb58931c-9eb3-4363-b340-5aa9e64ed348 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:45.695952 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb58931c-9eb3-4363-b340-5aa9e64ed348 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:45.697318 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb58931c-9eb3-4363-b340-5aa9e64ed348 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:45.710602 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 246/476] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:33:45 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:45.716041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-9615c563-fc7b-4b1a-8ae6-f64123c18638 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Aug 24 15:33:45.716041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:45.716041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 24 15:33:45.716041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 24 15:33:45.716041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 24 15:33:45.716041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 24 15:33:45.716041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 24 15:33:45.716041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application result = self._query(query) Aug 24 15:33:45.716041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 24 15:33:45.716041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application conn.query(q) Aug 24 15:33:45.716041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 24 15:33:45.716041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 24 15:33:45.716041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 24 15:33:45.716041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application result.read() Aug 24 15:33:45.716041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 24 15:33:45.717866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 24 15:33:45.717866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 24 15:33:45.717866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 24 15:33:45.717866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 24 15:33:45.717866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 24 15:33:45.717866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 24 15:33:45.717866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 24 15:33:45.717866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 24 15:33:45.717866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:45.717866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 24 15:33:45.717866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:45.717866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:45.717866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 24 15:33:45.717866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 24 15:33:45.717866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Aug 24 15:33:45.717866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return ref.to_dict() Aug 24 15:33:45.717866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 24 15:33:45.717866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application next(self.gen) Aug 24 15:33:45.719207 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 24 15:33:45.719207 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application yield resource Aug 24 15:33:45.719207 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 24 15:33:45.719207 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application next(self.gen) Aug 24 15:33:45.719207 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 24 15:33:45.719207 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application self.session.rollback() Aug 24 15:33:45.719207 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 24 15:33:45.719207 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application self.force_reraise() Aug 24 15:33:45.719207 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 24 15:33:45.719207 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise self.value Aug 24 15:33:45.719207 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 24 15:33:45.719207 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 24 15:33:45.719207 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 24 15:33:45.719207 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application session.commit() Aug 24 15:33:45.719207 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 24 15:33:45.719207 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 24 15:33:45.719207 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 24 15:33:45.721649 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application self._prepare_impl() Aug 24 15:33:45.721649 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 24 15:33:45.721649 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application self.session.flush() Aug 24 15:33:45.721649 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 24 15:33:45.721649 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application self._flush(objects) Aug 24 15:33:45.721649 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 24 15:33:45.721649 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 24 15:33:45.721649 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 24 15:33:45.721649 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application compat.raise_( Aug 24 15:33:45.721649 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 24 15:33:45.721649 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception Aug 24 15:33:45.721649 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 24 15:33:45.721649 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application flush_context.execute() Aug 24 15:33:45.721649 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 24 15:33:45.721649 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rec.execute(self) Aug 24 15:33:45.721649 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 24 15:33:45.721649 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 24 15:33:45.723862 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 24 15:33:45.723862 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 24 15:33:45.723862 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Aug 24 15:33:45.723862 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application c = connection._execute_20( Aug 24 15:33:45.723862 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 24 15:33:45.723862 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 24 15:33:45.723862 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 24 15:33:45.723862 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 24 15:33:45.723862 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 24 15:33:45.723862 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 24 15:33:45.723862 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 24 15:33:45.723862 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 24 15:33:45.723862 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 24 15:33:45.723862 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 24 15:33:45.723862 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 24 15:33:45.723862 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception Aug 24 15:33:45.725596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 24 15:33:45.725596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 24 15:33:45.725596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 24 15:33:45.725596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 24 15:33:45.725596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 24 15:33:45.725596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application result = self._query(query) Aug 24 15:33:45.725596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 24 15:33:45.725596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application conn.query(q) Aug 24 15:33:45.725596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 24 15:33:45.725596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 24 15:33:45.725596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 24 15:33:45.725596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application result.read() Aug 24 15:33:45.725596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 24 15:33:45.725596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 24 15:33:45.725596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 24 15:33:45.725596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 24 15:33:45.726607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 24 15:33:45.726607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 24 15:33:45.726607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 24 15:33:45.726607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 24 15:33:45.726607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 24 15:33:45.726607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Aug 24 15:33:45.726607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [parameters: {'id': 'd795512273704103b0fb77fe7c377c24', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Aug 24 15:33:45.726607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 24 15:33:45.726607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:45.726607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 15:33:45.726607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:45.726607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:45.726607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:45.726607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:45.726607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:45.726607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:45.733190 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:45.733190 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:45.733190 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:45.733190 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:45.733190 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:45.733190 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:45.733190 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Aug 24 15:33:45.733190 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Aug 24 15:33:45.733190 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:45.733190 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:45.733190 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Aug 24 15:33:45.733190 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Aug 24 15:33:45.733190 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 24 15:33:45.733190 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 24 15:33:45.733190 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Aug 24 15:33:45.733190 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:45.734083 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 231/477] 10.222.0.102 () {60 vars in 1136 bytes} [Wed Aug 24 15:33:45 2022] POST /identity/v3/roles => generated 144 bytes in 249 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:33:45.734156 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb58931c-9eb3-4363-b340-5aa9e64ed348 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:45.734226 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9615c563-fc7b-4b1a-8ae6-f64123c18638 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:45.788069 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:45.863214 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9615c563-fc7b-4b1a-8ae6-f64123c18638 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:45.908834 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0711081c-4f29-4399-8223-3e9714984e4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:45.909866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0711081c-4f29-4399-8223-3e9714984e4f None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:45.910345 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0711081c-4f29-4399-8223-3e9714984e4f None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:45.910835 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0711081c-4f29-4399-8223-3e9714984e4f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:45.927348 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0711081c-4f29-4399-8223-3e9714984e4f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:45.933329 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-0711081c-4f29-4399-8223-3e9714984e4f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 15:33:45.933329 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:45.933329 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:45.933329 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:45.933329 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:45.933329 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:45.933329 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:45.933329 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:45.933329 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:45.933329 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:45.933329 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:45.933329 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:45.933329 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:45.933329 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:45.933329 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:45.933329 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:45.934225 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:45.934225 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:45.934225 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:45.934225 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:45.934225 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:45.934225 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:45.934225 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:45.934225 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:45.934225 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:45.934225 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:45.934225 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:45.934225 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:45.934225 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:45.934225 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:45.934225 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:45.934225 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:45.934992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:45.934992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:45.934992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:45.934992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:45.934992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:45.934992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:45.934992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:45.934992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:45.934992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:45.934992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:45.934992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 15:33:45.934992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:45.938284 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 232/478] 10.222.0.102 () {56 vars in 1119 bytes} [Wed Aug 24 15:33:45 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:33:45.944225 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0711081c-4f29-4399-8223-3e9714984e4f None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:45.980602 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:45.998536 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb58931c-9eb3-4363-b340-5aa9e64ed348 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:46.055104 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0711081c-4f29-4399-8223-3e9714984e4f None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:46.068017 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:46.092685 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838fd732-4f18-49e7-9c54-f544f816f501 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:46.093386 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838fd732-4f18-49e7-9c54-f544f816f501 None admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:46.093670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838fd732-4f18-49e7-9c54-f544f816f501 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:46.093946 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838fd732-4f18-49e7-9c54-f544f816f501 None admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:46.094616 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838fd732-4f18-49e7-9c54-f544f816f501 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:46.096375 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838fd732-4f18-49e7-9c54-f544f816f501 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:46.097710 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f1f835-5c02-48cc-b3f9-9a3cdf1fefac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:46.098528 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f1f835-5c02-48cc-b3f9-9a3cdf1fefac None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:46.099013 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f1f835-5c02-48cc-b3f9-9a3cdf1fefac None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:46.099486 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f1f835-5c02-48cc-b3f9-9a3cdf1fefac None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:46.100379 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4f1f835-5c02-48cc-b3f9-9a3cdf1fefac None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:46.100880 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4f1f835-5c02-48cc-b3f9-9a3cdf1fefac None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:46.102432 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4f1f835-5c02-48cc-b3f9-9a3cdf1fefac None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:46.119675 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 233/479] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:33:45 2022] GET /identity/v3/roles?name=anotherrole => generated 336 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:33:46.135114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 247/480] 10.222.0.102 () {60 vars in 1137 bytes} [Wed Aug 24 15:33:45 2022] POST /identity/v3/users => generated 250 bytes in 416 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:33:47.330683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21f7c3f-1077-425b-b6cb-4a16bf667bf9 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:47.331195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21f7c3f-1077-425b-b6cb-4a16bf667bf9 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:47.331195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21f7c3f-1077-425b-b6cb-4a16bf667bf9 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:47.332878 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 234/481] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:47 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:47.338779 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a260d1b9-88af-4e8d-b9a9-1850f9ac0533 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:47.339254 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a260d1b9-88af-4e8d-b9a9-1850f9ac0533 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:47.339327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a260d1b9-88af-4e8d-b9a9-1850f9ac0533 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:47.358828 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8e0f6f-7ebf-4e42-b804-e95e96e2dda8 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:47.359162 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8e0f6f-7ebf-4e42-b804-e95e96e2dda8 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:47.359368 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8e0f6f-7ebf-4e42-b804-e95e96e2dda8 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:47.360989 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 235/482] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:47 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:47.365385 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-a260d1b9-88af-4e8d-b9a9-1850f9ac0533 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:47.366219 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2afbe5-476a-4712-8fe5-268b641ca4f4 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:47.366506 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2afbe5-476a-4712-8fe5-268b641ca4f4 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:47.366608 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-a260d1b9-88af-4e8d-b9a9-1850f9ac0533 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:47.366779 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2afbe5-476a-4712-8fe5-268b641ca4f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:47.385163 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-ef2afbe5-476a-4712-8fe5-268b641ca4f4 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:47.386404 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-ef2afbe5-476a-4712-8fe5-268b641ca4f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:47.473266 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 248/483] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:47 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 15:33:47.484003 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 236/484] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:47 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 15:33:47.833900 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38183ad7-e696-449a-8c85-bc2d6fb11788 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:47.834383 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38183ad7-e696-449a-8c85-bc2d6fb11788 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:47.834758 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38183ad7-e696-449a-8c85-bc2d6fb11788 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:47.836515 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 249/485] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:47 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:47.842621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d325cc-ede8-4e60-8cf2-6095dd9b2347 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:47.843029 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d325cc-ede8-4e60-8cf2-6095dd9b2347 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:47.843398 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d325cc-ede8-4e60-8cf2-6095dd9b2347 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:47.862212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-75d325cc-ede8-4e60-8cf2-6095dd9b2347 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:47.863390 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-75d325cc-ede8-4e60-8cf2-6095dd9b2347 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:47.877825 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38183ad7-e696-449a-8c85-bc2d6fb11788 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:47.960359 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 237/486] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:47 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:47.961227 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d325cc-ede8-4e60-8cf2-6095dd9b2347 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:47.996106 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:48.011979 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:48.025651 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38183ad7-e696-449a-8c85-bc2d6fb11788 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:48.094655 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82921595-ddc8-4a53-897c-e2f6550b9096 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:48.096468 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82921595-ddc8-4a53-897c-e2f6550b9096 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:48.097888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82921595-ddc8-4a53-897c-e2f6550b9096 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:48.098615 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82921595-ddc8-4a53-897c-e2f6550b9096 None admin] PATH_INFO: `/v3/roles/service` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:48.120176 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82921595-ddc8-4a53-897c-e2f6550b9096 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:48.127881 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-82921595-ddc8-4a53-897c-e2f6550b9096 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:33:48.127881 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:48.127881 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:48.127881 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:48.127881 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:48.127881 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:48.127881 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:48.127881 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:48.127881 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:48.127881 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:48.127881 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:48.127881 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:48.127881 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:48.127881 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:48.127881 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:48.127881 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:48.128788 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:48.128788 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:48.128788 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:48.128788 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:48.128788 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:48.128788 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:48.128788 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:48.128788 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:48.128788 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:48.128788 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:48.128788 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:48.128788 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:48.128788 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:48.128788 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:48.128788 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:48.128788 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:48.129563 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:48.129563 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:48.129563 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:48.129563 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:48.129563 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:48.129563 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:48.129563 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:48.129563 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:48.129563 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:48.129563 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:48.129563 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:33:48.129563 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:48.132279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 250/487] 10.222.0.102 () {56 vars in 1107 bytes} [Wed Aug 24 15:33:47 2022] GET /identity/v3/roles/service => generated 85 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:48.140211 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82921595-ddc8-4a53-897c-e2f6550b9096 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:48.166122 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d325cc-ede8-4e60-8cf2-6095dd9b2347 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:48.233928 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52aae9b9-c4cd-4c7a-a75f-6273cd97eed2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:48.236153 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52aae9b9-c4cd-4c7a-a75f-6273cd97eed2 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:48.236671 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52aae9b9-c4cd-4c7a-a75f-6273cd97eed2 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:48.237143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52aae9b9-c4cd-4c7a-a75f-6273cd97eed2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:48.257974 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52aae9b9-c4cd-4c7a-a75f-6273cd97eed2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:48.264139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-52aae9b9-c4cd-4c7a-a75f-6273cd97eed2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:33:48.264139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:48.264139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:48.264139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:48.264139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:48.264139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:48.264139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:48.264139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:48.264139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:48.264139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:48.264139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:48.264139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:48.264139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:48.264139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:48.264139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:48.264139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:48.265073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:48.265073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:48.265073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:48.265073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:48.265073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:48.265073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:48.265073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:48.265073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:48.265073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:48.265073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:48.265073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:48.265073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:48.265073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:48.265073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:48.265073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:48.265073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:48.265840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:48.265840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:48.265840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:48.265840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:48.265840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:48.265840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:48.265840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:48.265840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:48.265840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:48.265840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:48.265840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:33:48.265840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:48.273548 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 238/488] 10.222.0.102 () {56 vars in 1107 bytes} [Wed Aug 24 15:33:47 2022] GET /identity/v3/roles/service => generated 85 bytes in 312 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:48.280645 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52aae9b9-c4cd-4c7a-a75f-6273cd97eed2 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:48.335767 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82921595-ddc8-4a53-897c-e2f6550b9096 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:48.380901 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561e5ce9-0111-429f-ad65-e961ae1f1db2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:48.381540 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561e5ce9-0111-429f-ad65-e961ae1f1db2 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:48.381784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561e5ce9-0111-429f-ad65-e961ae1f1db2 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:48.382059 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561e5ce9-0111-429f-ad65-e961ae1f1db2 None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:48.383082 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561e5ce9-0111-429f-ad65-e961ae1f1db2 None admin] RBAC: Adding query filter params (name=service) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:48.383413 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561e5ce9-0111-429f-ad65-e961ae1f1db2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:48.385132 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561e5ce9-0111-429f-ad65-e961ae1f1db2 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:48.398320 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 251/489] 10.222.0.102 () {56 vars in 1108 bytes} [Wed Aug 24 15:33:48 2022] GET /identity/v3/roles?name=service => generated 328 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:48.399681 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561e5ce9-0111-429f-ad65-e961ae1f1db2 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:48.403573 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:48.434480 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:48.455395 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52aae9b9-c4cd-4c7a-a75f-6273cd97eed2 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:48.491549 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9f389a-e05a-4c51-8401-fcf38279d5a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:48.492184 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9f389a-e05a-4c51-8401-fcf38279d5a0 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:48.492561 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9f389a-e05a-4c51-8401-fcf38279d5a0 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:48.492935 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9f389a-e05a-4c51-8401-fcf38279d5a0 None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:48.493599 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c9f389a-e05a-4c51-8401-fcf38279d5a0 None admin] RBAC: Adding query filter params (name=service) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:48.493910 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c9f389a-e05a-4c51-8401-fcf38279d5a0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:48.495391 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c9f389a-e05a-4c51-8401-fcf38279d5a0 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:48.504371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 239/490] 10.222.0.102 () {56 vars in 1108 bytes} [Wed Aug 24 15:33:48 2022] GET /identity/v3/roles?name=service => generated 328 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:33:48.505850 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9f389a-e05a-4c51-8401-fcf38279d5a0 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:48.535708 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561e5ce9-0111-429f-ad65-e961ae1f1db2 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:48.574538 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065c0bab-066d-4ad5-950e-ec0ee9a1fc6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:48.575538 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065c0bab-066d-4ad5-950e-ec0ee9a1fc6f None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:48.575999 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065c0bab-066d-4ad5-950e-ec0ee9a1fc6f None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:48.576912 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065c0bab-066d-4ad5-950e-ec0ee9a1fc6f None admin] PATH_INFO: `/v3/roles/service` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:48.588500 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065c0bab-066d-4ad5-950e-ec0ee9a1fc6f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:48.594791 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-065c0bab-066d-4ad5-950e-ec0ee9a1fc6f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:33:48.594791 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:48.594791 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:48.594791 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:48.594791 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:48.594791 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:48.594791 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:48.594791 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:48.594791 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:48.594791 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:48.594791 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:48.594791 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:48.594791 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:48.594791 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:48.594791 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:48.594791 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:48.595858 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:48.595858 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:48.595858 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:48.595858 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:48.595858 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:48.595858 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:48.595858 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:48.595858 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:48.595858 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:48.595858 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:48.595858 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:48.595858 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:48.595858 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:48.595858 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:48.595858 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:48.595858 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:48.596665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:48.596665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:48.596665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:48.596665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:48.596665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:48.596665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:48.596665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:48.596665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:48.596665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:48.596665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:48.596665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:33:48.596665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:48.597257 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 252/491] 10.222.0.102 () {56 vars in 1107 bytes} [Wed Aug 24 15:33:48 2022] GET /identity/v3/roles/service => generated 85 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:48.598112 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0355b1-2188-4527-97a9-870986e7492e None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:48.598394 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0355b1-2188-4527-97a9-870986e7492e None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:48.598667 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0355b1-2188-4527-97a9-870986e7492e None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:48.600016 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 253/492] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:48 2022] GET /identity => generated 271 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:48.604682 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0355b1-2188-4527-97a9-870986e7492e None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:48.625496 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9f389a-e05a-4c51-8401-fcf38279d5a0 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:48.666848 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b167b74-abe1-4dc6-9228-b93120370d06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:48.667488 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b167b74-abe1-4dc6-9228-b93120370d06 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:48.667827 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b167b74-abe1-4dc6-9228-b93120370d06 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:48.668109 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b167b74-abe1-4dc6-9228-b93120370d06 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:48.679940 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:48.681215 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b167b74-abe1-4dc6-9228-b93120370d06 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:48.683154 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b167b74-abe1-4dc6-9228-b93120370d06 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:48.701759 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-8b167b74-abe1-4dc6-9228-b93120370d06 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:48.701759 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:48.701759 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:48.701759 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:48.701759 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:48.701759 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:48.701759 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:48.701759 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:48.701759 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:48.701759 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:48.701759 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:48.701759 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:48.701759 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:33:48.701759 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:33:48.701759 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:33:48.701759 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:33:48.708118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:48.708118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:48.708118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:48.708118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:48.708118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:48.708118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:48.708118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:48.708118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:48.708118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:48.708118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:48.708118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:48.708118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:48.708118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:48.708118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:48.708118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:48.708118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:48.708118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:33:48.710345 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:33:48.710345 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:48.710345 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:48.710345 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 240/493] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:33:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:48.710345 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b167b74-abe1-4dc6-9228-b93120370d06 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:48.710345 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:48.734983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0355b1-2188-4527-97a9-870986e7492e None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:48.771237 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d825d9bb-e668-41b5-a4a3-89d94176abe2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:48.771904 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d825d9bb-e668-41b5-a4a3-89d94176abe2 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:48.772186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d825d9bb-e668-41b5-a4a3-89d94176abe2 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:48.772461 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d825d9bb-e668-41b5-a4a3-89d94176abe2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:48.784427 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d825d9bb-e668-41b5-a4a3-89d94176abe2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:48.786112 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d825d9bb-e668-41b5-a4a3-89d94176abe2 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:48.802420 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-d825d9bb-e668-41b5-a4a3-89d94176abe2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:48.802420 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:48.802420 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:48.802420 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:48.802420 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:48.802420 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:48.802420 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:48.802420 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:48.802420 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:48.802420 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:48.802420 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:48.802420 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:48.802420 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:33:48.802420 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:33:48.802420 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:33:48.802420 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:33:48.803680 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:48.803680 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:48.803680 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:48.803680 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:48.803680 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:48.803680 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:48.803680 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:48.803680 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:48.803680 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:48.803680 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:48.803680 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:48.803680 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:48.803680 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:48.803680 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:48.803680 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:48.803680 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:48.803680 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:33:48.804687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:33:48.804687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:48.804687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:48.807778 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 254/494] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:33:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:48.809354 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3f4a33-ccc7-43fa-ae01-cb119381fcb1 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:48.809648 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3f4a33-ccc7-43fa-ae01-cb119381fcb1 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:48.809928 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3f4a33-ccc7-43fa-ae01-cb119381fcb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:48.829870 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b167b74-abe1-4dc6-9228-b93120370d06 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:48.833418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-ca3f4a33-ccc7-43fa-ae01-cb119381fcb1 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:48.834645 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-ca3f4a33-ccc7-43fa-ae01-cb119381fcb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:48.865879 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249eb917-a498-4f40-8d81-24fd10cb6bad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:48.866487 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249eb917-a498-4f40-8d81-24fd10cb6bad None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:48.866763 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249eb917-a498-4f40-8d81-24fd10cb6bad None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:48.867036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249eb917-a498-4f40-8d81-24fd10cb6bad None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:48.867736 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249eb917-a498-4f40-8d81-24fd10cb6bad None admin] RBAC: Adding query filter params (name=service) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:48.868044 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249eb917-a498-4f40-8d81-24fd10cb6bad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:48.869431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249eb917-a498-4f40-8d81-24fd10cb6bad None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:48.878044 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 241/495] 10.222.0.102 () {56 vars in 1108 bytes} [Wed Aug 24 15:33:48 2022] GET /identity/v3/roles?name=service => generated 328 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:48.878873 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249eb917-a498-4f40-8d81-24fd10cb6bad None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:48.912143 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 255/496] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:48 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:48.913417 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04e1c9a-9169-4739-ae6e-cb56e170dd47 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:48.913712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04e1c9a-9169-4739-ae6e-cb56e170dd47 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:48.913992 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04e1c9a-9169-4739-ae6e-cb56e170dd47 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:48.915440 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 256/497] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:48 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:48.916039 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04e1c9a-9169-4739-ae6e-cb56e170dd47 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:48.946307 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:48.974908 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:48.993568 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249eb917-a498-4f40-8d81-24fd10cb6bad None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:49.032718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b73f58-4d20-4f61-bfd6-2a9c0d15fe84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:49.033572 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b73f58-4d20-4f61-bfd6-2a9c0d15fe84 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:49.033853 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b73f58-4d20-4f61-bfd6-2a9c0d15fe84 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:49.034269 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b73f58-4d20-4f61-bfd6-2a9c0d15fe84 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:49.035234 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b73f58-4d20-4f61-bfd6-2a9c0d15fe84 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:49.035691 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b73f58-4d20-4f61-bfd6-2a9c0d15fe84 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:49.037292 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b73f58-4d20-4f61-bfd6-2a9c0d15fe84 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:49.038540 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04e1c9a-9169-4739-ae6e-cb56e170dd47 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:49.056564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 242/498] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:33:48 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:49.057443 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b73f58-4d20-4f61-bfd6-2a9c0d15fe84 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:49.078967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0baf510c-d70b-4a2e-8960-445de96ac67a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:49.079580 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0baf510c-d70b-4a2e-8960-445de96ac67a None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:49.079910 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0baf510c-d70b-4a2e-8960-445de96ac67a None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:49.080185 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0baf510c-d70b-4a2e-8960-445de96ac67a None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:49.080837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0baf510c-d70b-4a2e-8960-445de96ac67a None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:49.081141 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0baf510c-d70b-4a2e-8960-445de96ac67a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:49.082521 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0baf510c-d70b-4a2e-8960-445de96ac67a None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:49.095620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 257/499] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:33:48 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:49.096847 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3690ca60-1290-490f-9afd-4509483e93c1 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:49.097142 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3690ca60-1290-490f-9afd-4509483e93c1 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:49.097418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3690ca60-1290-490f-9afd-4509483e93c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:49.116729 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-3690ca60-1290-490f-9afd-4509483e93c1 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:49.117924 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-3690ca60-1290-490f-9afd-4509483e93c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:49.183576 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b73f58-4d20-4f61-bfd6-2a9c0d15fe84 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:49.189631 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 258/500] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:49 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:49.190488 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3690ca60-1290-490f-9afd-4509483e93c1 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:49.215100 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:49.228677 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10028440-e570-4e28-97c1-54a2b2d9c624 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:49.229309 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10028440-e570-4e28-97c1-54a2b2d9c624 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:49.229592 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10028440-e570-4e28-97c1-54a2b2d9c624 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:49.229863 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10028440-e570-4e28-97c1-54a2b2d9c624 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:49.231264 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:49.242463 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10028440-e570-4e28-97c1-54a2b2d9c624 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:49.244011 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10028440-e570-4e28-97c1-54a2b2d9c624 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:49.260582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-10028440-e570-4e28-97c1-54a2b2d9c624 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:49.260582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:49.260582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:49.260582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:49.260582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:49.260582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:49.260582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:49.260582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:49.260582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:49.260582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:49.260582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:49.260582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:49.260582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:33:49.260582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:33:49.260582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:33:49.260582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:33:49.261716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:49.261716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:49.261716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:49.261716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:49.261716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:49.261716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:49.261716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:49.261716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:49.261716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:49.261716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:49.261716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:49.261716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:49.261716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:49.261716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:49.261716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:49.261716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:49.261716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:33:49.262648 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:33:49.262648 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:49.262648 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:49.262648 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 243/501] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:33:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:49.262648 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10028440-e570-4e28-97c1-54a2b2d9c624 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:49.319271 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3690ca60-1290-490f-9afd-4509483e93c1 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:49.356534 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bdafaeb-fec9-462c-936e-5f3c11a3d2b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:49.357172 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bdafaeb-fec9-462c-936e-5f3c11a3d2b7 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:49.357545 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bdafaeb-fec9-462c-936e-5f3c11a3d2b7 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:49.357889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bdafaeb-fec9-462c-936e-5f3c11a3d2b7 None admin] PATH_INFO: `/v3/users/glance` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:49.365292 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bdafaeb-fec9-462c-936e-5f3c11a3d2b7 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:49.366750 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bdafaeb-fec9-462c-936e-5f3c11a3d2b7 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:49.381609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-5bdafaeb-fec9-462c-936e-5f3c11a3d2b7 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 24 15:33:49.381609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:49.381609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:49.381609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:49.381609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:49.381609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:49.381609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:49.381609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:49.381609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:49.381609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:49.381609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:49.381609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:49.381609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:33:49.381609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:33:49.381609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:33:49.381609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:33:49.389754 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:49.389754 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:49.389754 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:33:49.389754 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:33:49.389754 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:33:49.389754 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:33:49.389754 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:49.389754 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:49.389754 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:49.389754 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:49.389754 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:49.389754 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:49.389754 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:49.389754 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:49.389754 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:49.389754 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:49.389754 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:49.389754 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:49.390784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:49.390784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:49.390784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:33:49.390784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:33:49.390784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:33:49.390784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:33:49.390784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:33:49.390784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:33:49.390784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 24 15:33:49.390784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:49.391366 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10028440-e570-4e28-97c1-54a2b2d9c624 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:49.391457 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 259/502] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:33:49 2022] GET /identity/v3/users/glance => generated 84 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:49.391457 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bdafaeb-fec9-462c-936e-5f3c11a3d2b7 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:49.429812 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ddea52-37d8-43a1-b3aa-8b415a739e13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:49.430619 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ddea52-37d8-43a1-b3aa-8b415a739e13 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:49.430989 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ddea52-37d8-43a1-b3aa-8b415a739e13 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:49.431362 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ddea52-37d8-43a1-b3aa-8b415a739e13 None admin] PATH_INFO: `/v3/users/nova` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:49.440704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ddea52-37d8-43a1-b3aa-8b415a739e13 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:49.442148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ddea52-37d8-43a1-b3aa-8b415a739e13 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:49.455433 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-42ddea52-37d8-43a1-b3aa-8b415a739e13 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 24 15:33:49.455433 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:49.455433 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:49.455433 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:49.455433 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:49.455433 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:49.455433 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:49.455433 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:49.455433 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:49.455433 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:49.455433 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:49.455433 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:49.455433 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:33:49.455433 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:33:49.455433 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:33:49.455433 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:33:49.456609 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:49.456609 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:49.456609 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:33:49.456609 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:33:49.456609 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:33:49.456609 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:33:49.456609 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:49.456609 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:49.456609 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:49.456609 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:49.456609 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:49.456609 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:49.456609 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:49.456609 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:49.456609 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:49.456609 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:49.456609 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:49.456609 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:49.457705 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:49.457705 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:49.457705 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:33:49.457705 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:33:49.457705 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:33:49.457705 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:33:49.457705 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:33:49.457705 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:33:49.457705 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 24 15:33:49.457705 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:49.461222 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 244/503] 10.222.0.102 () {56 vars in 1098 bytes} [Wed Aug 24 15:33:49 2022] GET /identity/v3/users/nova => generated 82 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:49.462126 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ddea52-37d8-43a1-b3aa-8b415a739e13 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:49.497400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:49.497572 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:49.507946 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bdafaeb-fec9-462c-936e-5f3c11a3d2b7 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:49.552380 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd98d0f9-90ff-4d2d-9a73-7ec984df42b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:49.553008 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd98d0f9-90ff-4d2d-9a73-7ec984df42b4 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:49.553284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd98d0f9-90ff-4d2d-9a73-7ec984df42b4 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:49.553553 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd98d0f9-90ff-4d2d-9a73-7ec984df42b4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:49.565065 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd98d0f9-90ff-4d2d-9a73-7ec984df42b4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:49.566554 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd98d0f9-90ff-4d2d-9a73-7ec984df42b4 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:49.574120 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ddea52-37d8-43a1-b3aa-8b415a739e13 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:49.581583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-cd98d0f9-90ff-4d2d-9a73-7ec984df42b4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:49.581583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:49.581583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:49.581583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:49.581583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:49.581583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:49.581583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:49.581583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:49.581583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:49.581583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:49.581583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:49.581583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:49.581583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:33:49.581583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:33:49.581583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:33:49.581583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:33:49.582712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:49.582712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:49.582712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:49.582712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:49.582712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:49.582712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:49.582712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:49.582712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:49.582712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:49.582712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:49.582712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:49.582712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:49.582712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:49.582712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:49.582712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:49.582712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:49.582712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:33:49.583549 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:33:49.583549 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:49.583549 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:49.583549 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 260/504] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:33:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:49.583549 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd98d0f9-90ff-4d2d-9a73-7ec984df42b4 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:49.617560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a447e1-055b-416b-8f35-eebfde84af94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:49.618165 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a447e1-055b-416b-8f35-eebfde84af94 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:49.618476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a447e1-055b-416b-8f35-eebfde84af94 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:49.618729 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a447e1-055b-416b-8f35-eebfde84af94 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:49.619375 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a447e1-055b-416b-8f35-eebfde84af94 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:49.619719 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a447e1-055b-416b-8f35-eebfde84af94 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:49.621196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a447e1-055b-416b-8f35-eebfde84af94 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:49.633799 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 245/505] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:33:49 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:49.633799 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a447e1-055b-416b-8f35-eebfde84af94 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:49.689572 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd98d0f9-90ff-4d2d-9a73-7ec984df42b4 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:49.726366 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ee847d-c705-4263-bf13-436140b300fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:49.726988 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ee847d-c705-4263-bf13-436140b300fd None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:49.727265 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ee847d-c705-4263-bf13-436140b300fd None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:49.727536 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ee847d-c705-4263-bf13-436140b300fd None admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:49.728302 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ee847d-c705-4263-bf13-436140b300fd None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:49.728606 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ee847d-c705-4263-bf13-436140b300fd None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:49.730048 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ee847d-c705-4263-bf13-436140b300fd None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:49.735389 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:49.742520 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a447e1-055b-416b-8f35-eebfde84af94 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:49.772251 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 261/506] 10.222.0.102 () {56 vars in 1142 bytes} [Wed Aug 24 15:33:49 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 374 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:49.772251 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ee847d-c705-4263-bf13-436140b300fd None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:49.773604 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:49.781021 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f894bc-c571-441e-86bd-26511b7cfc0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:49.781658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f894bc-c571-441e-86bd-26511b7cfc0b None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:49.781943 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f894bc-c571-441e-86bd-26511b7cfc0b None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:49.782218 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f894bc-c571-441e-86bd-26511b7cfc0b None admin] PATH_INFO: `/v3/roles/service` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:49.792977 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f894bc-c571-441e-86bd-26511b7cfc0b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:49.798415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-79f894bc-c571-441e-86bd-26511b7cfc0b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:33:49.798415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:49.798415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:49.798415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:49.798415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:49.798415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:49.798415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:49.798415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:49.798415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:49.798415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:49.798415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:49.798415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:49.798415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:49.798415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:49.798415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:49.798415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:49.803980 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:49.803980 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:49.803980 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:49.803980 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:49.803980 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:49.803980 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:49.803980 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:49.803980 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:49.803980 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:49.803980 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:49.803980 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:49.803980 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:49.803980 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:49.803980 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:49.803980 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:49.803980 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:49.804811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:49.804811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:49.804811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:49.804811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:49.804811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:49.804811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:49.804811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:49.804811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:49.804811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:49.804811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:49.804811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:33:49.804811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:49.804811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 246/507] 10.222.0.102 () {56 vars in 1107 bytes} [Wed Aug 24 15:33:49 2022] GET /identity/v3/roles/service => generated 85 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:49.804811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f894bc-c571-441e-86bd-26511b7cfc0b None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:49.880296 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ee847d-c705-4263-bf13-436140b300fd None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:49.921056 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f223aa3-389b-4043-848c-909bedaf1e45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:49.921315 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f894bc-c571-441e-86bd-26511b7cfc0b None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:49.923354 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f223aa3-389b-4043-848c-909bedaf1e45 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:49.923354 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f223aa3-389b-4043-848c-909bedaf1e45 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:49.923354 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f223aa3-389b-4043-848c-909bedaf1e45 None admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:49.923354 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f223aa3-389b-4043-848c-909bedaf1e45 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:49.923354 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f223aa3-389b-4043-848c-909bedaf1e45 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:49.924694 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f223aa3-389b-4043-848c-909bedaf1e45 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:49.942837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 262/508] 10.222.0.102 () {56 vars in 1138 bytes} [Wed Aug 24 15:33:49 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 370 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:49.943679 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f223aa3-389b-4043-848c-909bedaf1e45 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:49.962636 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f5be66-1c65-4923-a809-73b81c556a2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:49.963247 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f5be66-1c65-4923-a809-73b81c556a2e None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:49.963524 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f5be66-1c65-4923-a809-73b81c556a2e None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:49.963828 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f5be66-1c65-4923-a809-73b81c556a2e None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:49.964490 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f5be66-1c65-4923-a809-73b81c556a2e None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:49.964795 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f5be66-1c65-4923-a809-73b81c556a2e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:49.966158 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f5be66-1c65-4923-a809-73b81c556a2e None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:49.983745 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 247/509] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:33:49 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:49.983745 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f5be66-1c65-4923-a809-73b81c556a2e None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:50.002845 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:50.011251 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:50.052291 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f223aa3-389b-4043-848c-909bedaf1e45 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:50.087467 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029d3f83-d72d-4d77-a320-c882bcf76c76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:50.087896 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f5be66-1c65-4923-a809-73b81c556a2e None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:50.088124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029d3f83-d72d-4d77-a320-c882bcf76c76 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:50.088402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029d3f83-d72d-4d77-a320-c882bcf76c76 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:50.088673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029d3f83-d72d-4d77-a320-c882bcf76c76 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:50.096671 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029d3f83-d72d-4d77-a320-c882bcf76c76 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:50.098465 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029d3f83-d72d-4d77-a320-c882bcf76c76 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:50.111744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-029d3f83-d72d-4d77-a320-c882bcf76c76 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 24 15:33:50.111744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:50.111744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:50.111744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:50.111744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:50.111744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:50.111744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:50.111744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:50.111744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:50.111744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:50.111744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:50.111744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:50.111744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:33:50.111744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:33:50.111744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:33:50.111744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:33:50.112864 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:50.112864 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:50.112864 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:33:50.112864 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:33:50.112864 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:33:50.112864 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:33:50.112864 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:50.112864 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:50.112864 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:50.112864 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:50.112864 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:50.112864 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:50.112864 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:50.112864 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:50.112864 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:50.112864 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:50.112864 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:50.112864 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:50.113889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:50.113889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:50.113889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:33:50.113889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:33:50.113889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:33:50.113889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:33:50.113889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:33:50.113889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:33:50.113889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 24 15:33:50.113889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:50.113889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 263/510] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:33:49 2022] GET /identity/v3/users/cinder => generated 84 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:50.113889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029d3f83-d72d-4d77-a320-c882bcf76c76 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:50.126357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63388be3-de0f-46d6-a4b9-3787ed8e3e00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:50.126972 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63388be3-de0f-46d6-a4b9-3787ed8e3e00 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:50.127250 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63388be3-de0f-46d6-a4b9-3787ed8e3e00 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:50.127525 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63388be3-de0f-46d6-a4b9-3787ed8e3e00 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:50.195718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63388be3-de0f-46d6-a4b9-3787ed8e3e00 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:50.197478 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63388be3-de0f-46d6-a4b9-3787ed8e3e00 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:50.214198 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029d3f83-d72d-4d77-a320-c882bcf76c76 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:50.215188 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:50.238040 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:50.250391 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ebcd5c-ee0a-40e8-964c-3cc3a836ebf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:50.251000 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ebcd5c-ee0a-40e8-964c-3cc3a836ebf5 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:50.251275 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ebcd5c-ee0a-40e8-964c-3cc3a836ebf5 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:50.251548 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ebcd5c-ee0a-40e8-964c-3cc3a836ebf5 None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:50.252245 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ebcd5c-ee0a-40e8-964c-3cc3a836ebf5 None admin] RBAC: Adding query filter params (name=service) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:50.252546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ebcd5c-ee0a-40e8-964c-3cc3a836ebf5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:50.253909 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ebcd5c-ee0a-40e8-964c-3cc3a836ebf5 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:50.262028 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 264/511] 10.222.0.102 () {56 vars in 1108 bytes} [Wed Aug 24 15:33:50 2022] GET /identity/v3/roles?name=service => generated 328 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:50.262830 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ebcd5c-ee0a-40e8-964c-3cc3a836ebf5 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:50.306972 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 248/512] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:33:49 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:33:50.307740 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63388be3-de0f-46d6-a4b9-3787ed8e3e00 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:50.375305 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ebcd5c-ee0a-40e8-964c-3cc3a836ebf5 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:50.410402 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63388be3-de0f-46d6-a4b9-3787ed8e3e00 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:50.411984 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb697d6-c5c3-4d70-9ac1-447bac6db169 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:50.412674 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb697d6-c5c3-4d70-9ac1-447bac6db169 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:50.412886 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb697d6-c5c3-4d70-9ac1-447bac6db169 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:50.413159 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb697d6-c5c3-4d70-9ac1-447bac6db169 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:50.427936 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:50.448769 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f1de29-07e2-4f06-aa96-f64f7d7a61cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:50.449621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f1de29-07e2-4f06-aa96-f64f7d7a61cd None admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:50.449621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f1de29-07e2-4f06-aa96-f64f7d7a61cd None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:50.449904 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f1de29-07e2-4f06-aa96-f64f7d7a61cd None admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:50.450555 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f1de29-07e2-4f06-aa96-f64f7d7a61cd None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:50.452607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f1de29-07e2-4f06-aa96-f64f7d7a61cd None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:50.470096 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:50.479679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 249/513] 10.222.0.102 () {60 vars in 1137 bytes} [Wed Aug 24 15:33:50 2022] POST /identity/v3/users => generated 248 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:33:50.479679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f1de29-07e2-4f06-aa96-f64f7d7a61cd None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:50.499668 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb697d6-c5c3-4d70-9ac1-447bac6db169 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:50.501500 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb697d6-c5c3-4d70-9ac1-447bac6db169 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:50.601604 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f1de29-07e2-4f06-aa96-f64f7d7a61cd None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:50.627137 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 265/514] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:33:50 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:33:50.627761 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb697d6-c5c3-4d70-9ac1-447bac6db169 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:50.645991 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b6c67b-137a-4402-860f-08d847d18128 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:50.647015 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b6c67b-137a-4402-860f-08d847d18128 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:50.647015 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b6c67b-137a-4402-860f-08d847d18128 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:50.647198 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b6c67b-137a-4402-860f-08d847d18128 None admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:50.648616 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b6c67b-137a-4402-860f-08d847d18128 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:50.648616 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b6c67b-137a-4402-860f-08d847d18128 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:50.649769 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b6c67b-137a-4402-860f-08d847d18128 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:50.694555 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 250/515] 10.222.0.102 () {56 vars in 1142 bytes} [Wed Aug 24 15:33:50 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 374 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:50.695450 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b6c67b-137a-4402-860f-08d847d18128 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:50.729382 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:50.731721 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:50.750864 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb697d6-c5c3-4d70-9ac1-447bac6db169 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:50.791199 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123a29d7-51c9-4cc6-839d-bcf1008c2360 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:50.791870 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123a29d7-51c9-4cc6-839d-bcf1008c2360 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:50.792196 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123a29d7-51c9-4cc6-839d-bcf1008c2360 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:50.792474 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123a29d7-51c9-4cc6-839d-bcf1008c2360 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:50.803000 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123a29d7-51c9-4cc6-839d-bcf1008c2360 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:50.804507 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123a29d7-51c9-4cc6-839d-bcf1008c2360 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:50.815035 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b6c67b-137a-4402-860f-08d847d18128 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:50.819938 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-123a29d7-51c9-4cc6-839d-bcf1008c2360 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:50.819938 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:50.819938 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:50.819938 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:50.819938 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:50.819938 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:50.819938 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:50.819938 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:50.819938 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:50.819938 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:50.819938 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:50.819938 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:50.819938 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:33:50.819938 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:33:50.819938 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:33:50.819938 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:33:50.820826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:50.820826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:50.820826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:50.820826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:50.820826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:50.820826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:50.820826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:50.820826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:50.820826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:50.820826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:50.820826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:50.820826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:50.820826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:50.820826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:50.820826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:50.820826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:50.820826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:33:50.821669 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:33:50.821669 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:50.821669 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:50.821669 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 266/516] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:33:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:50.827954 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123a29d7-51c9-4cc6-839d-bcf1008c2360 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:50.861905 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c79868-6f30-412d-9c02-6f6eb73534f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:50.862455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c79868-6f30-412d-9c02-6f6eb73534f2 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:50.862761 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c79868-6f30-412d-9c02-6f6eb73534f2 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:50.863033 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c79868-6f30-412d-9c02-6f6eb73534f2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:50.875107 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c79868-6f30-412d-9c02-6f6eb73534f2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:50.876792 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c79868-6f30-412d-9c02-6f6eb73534f2 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:50.893718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-b8c79868-6f30-412d-9c02-6f6eb73534f2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:50.893718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:50.893718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:50.893718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:50.893718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:50.893718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:50.893718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:50.893718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:50.893718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:50.893718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:50.893718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:50.893718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:50.893718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:33:50.893718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:33:50.893718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:33:50.893718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:33:50.894664 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:50.894664 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:50.894664 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:50.894664 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:50.894664 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:50.894664 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:50.894664 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:50.894664 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:50.894664 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:50.894664 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:50.894664 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:50.894664 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:50.894664 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:50.894664 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:50.894664 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:50.894664 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:50.894664 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:33:50.895492 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:33:50.895492 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:50.895492 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:50.895492 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 251/517] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:33:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:50.896193 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c79868-6f30-412d-9c02-6f6eb73534f2 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:50.947248 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123a29d7-51c9-4cc6-839d-bcf1008c2360 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:50.985687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd32172-270c-44f0-ac52-b41e5c93bd83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:50.986334 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd32172-270c-44f0-ac52-b41e5c93bd83 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:50.986618 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd32172-270c-44f0-ac52-b41e5c93bd83 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:50.986890 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd32172-270c-44f0-ac52-b41e5c93bd83 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:50.989840 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:50.990739 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:50.999463 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd32172-270c-44f0-ac52-b41e5c93bd83 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:51.000976 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd32172-270c-44f0-ac52-b41e5c93bd83 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:51.012713 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c79868-6f30-412d-9c02-6f6eb73534f2 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:51.016824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-2fd32172-270c-44f0-ac52-b41e5c93bd83 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:51.016824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:51.016824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:51.016824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:51.016824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:51.016824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:51.016824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:51.016824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:51.016824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:51.016824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:51.016824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:51.016824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:51.016824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:33:51.016824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:33:51.016824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:33:51.016824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:33:51.017734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:51.017734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:51.017734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:51.017734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:51.017734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:51.017734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:51.017734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:51.017734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:51.017734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:51.017734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:51.017734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:51.017734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:51.017734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:51.017734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:51.017734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:51.017734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:51.017734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:33:51.018568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:33:51.018568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:51.018568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:51.022598 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 267/518] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:33:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:51.023487 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd32172-270c-44f0-ac52-b41e5c93bd83 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:51.050771 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6bc6e7-49a4-41e5-ae98-4e2d9a7ec339 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:51.051367 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6bc6e7-49a4-41e5-ae98-4e2d9a7ec339 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:51.051666 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6bc6e7-49a4-41e5-ae98-4e2d9a7ec339 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:51.051950 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6bc6e7-49a4-41e5-ae98-4e2d9a7ec339 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:51.132941 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6bc6e7-49a4-41e5-ae98-4e2d9a7ec339 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:51.135181 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6bc6e7-49a4-41e5-ae98-4e2d9a7ec339 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:51.143022 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd32172-270c-44f0-ac52-b41e5c93bd83 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:51.180417 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ab3988-24dc-457e-aa2d-b373c569a23d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:51.181031 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ab3988-24dc-457e-aa2d-b373c569a23d None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:51.181314 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ab3988-24dc-457e-aa2d-b373c569a23d None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:51.181590 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ab3988-24dc-457e-aa2d-b373c569a23d None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:51.182238 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ab3988-24dc-457e-aa2d-b373c569a23d None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:51.182542 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ab3988-24dc-457e-aa2d-b373c569a23d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:51.183942 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ab3988-24dc-457e-aa2d-b373c569a23d None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:51.196438 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 268/519] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:33:51 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:51.197262 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ab3988-24dc-457e-aa2d-b373c569a23d None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:51.235539 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:51.253302 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:51.264486 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 252/520] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:33:50 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:33:51.265682 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6bc6e7-49a4-41e5-ae98-4e2d9a7ec339 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:51.318405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ab3988-24dc-457e-aa2d-b373c569a23d None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:51.358634 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bca762c-f051-44f9-a6ac-96bbd40ea56c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:51.359694 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bca762c-f051-44f9-a6ac-96bbd40ea56c None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:51.360150 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bca762c-f051-44f9-a6ac-96bbd40ea56c None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:51.360523 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bca762c-f051-44f9-a6ac-96bbd40ea56c None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:51.361605 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bca762c-f051-44f9-a6ac-96bbd40ea56c None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:51.362112 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bca762c-f051-44f9-a6ac-96bbd40ea56c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:51.364524 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bca762c-f051-44f9-a6ac-96bbd40ea56c None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:51.382443 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6bc6e7-49a4-41e5-ae98-4e2d9a7ec339 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:51.387260 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 269/521] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:33:51 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:51.387260 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bca762c-f051-44f9-a6ac-96bbd40ea56c None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:51.427949 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ad32c8-4c08-4654-81ee-5703c918583b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:51.428648 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ad32c8-4c08-4654-81ee-5703c918583b None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:51.428996 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ad32c8-4c08-4654-81ee-5703c918583b None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:51.429337 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ad32c8-4c08-4654-81ee-5703c918583b None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:51.430130 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ad32c8-4c08-4654-81ee-5703c918583b None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:51.430496 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ad32c8-4c08-4654-81ee-5703c918583b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:51.431960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ad32c8-4c08-4654-81ee-5703c918583b None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:51.449597 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 253/522] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:33:51 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:51.449597 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ad32c8-4c08-4654-81ee-5703c918583b None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:51.501260 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:51.504501 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bca762c-f051-44f9-a6ac-96bbd40ea56c None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:51.515369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:51.542569 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ed0ae8-0a35-47e4-ac20-dd2e1c031617 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:51.543255 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ed0ae8-0a35-47e4-ac20-dd2e1c031617 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:51.543609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ed0ae8-0a35-47e4-ac20-dd2e1c031617 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:51.543983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ed0ae8-0a35-47e4-ac20-dd2e1c031617 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:51.552607 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ed0ae8-0a35-47e4-ac20-dd2e1c031617 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:51.554091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ed0ae8-0a35-47e4-ac20-dd2e1c031617 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:51.566327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-49ed0ae8-0a35-47e4-ac20-dd2e1c031617 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 24 15:33:51.566327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:51.566327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:51.566327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:51.566327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:51.566327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:51.566327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:51.566327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:51.566327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:51.566327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:51.566327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:51.566327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:51.566327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:33:51.566327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:33:51.566327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:33:51.566327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:33:51.567264 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:51.567264 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:51.567264 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:33:51.567264 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:33:51.567264 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:33:51.567264 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:33:51.567264 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:51.567264 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:51.567264 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:51.567264 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:51.567264 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:51.567264 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:51.567264 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:51.567264 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:51.567264 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:51.567264 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:51.567264 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:51.567264 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:51.568185 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:51.568185 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:51.568185 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:33:51.568185 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:33:51.568185 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:33:51.568185 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:33:51.568185 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:33:51.568185 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:33:51.568185 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 24 15:33:51.568185 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:51.569229 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ad32c8-4c08-4654-81ee-5703c918583b None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:51.572192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 270/523] 10.222.0.102 () {56 vars in 1107 bytes} [Wed Aug 24 15:33:51 2022] GET /identity/v3/users/neutron => generated 85 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:51.573106 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ed0ae8-0a35-47e4-ac20-dd2e1c031617 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:51.605386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc365fee-37db-4b16-b62f-65e483cf5494 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:51.606011 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc365fee-37db-4b16-b62f-65e483cf5494 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:51.606287 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc365fee-37db-4b16-b62f-65e483cf5494 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:51.606554 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc365fee-37db-4b16-b62f-65e483cf5494 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:51.617931 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc365fee-37db-4b16-b62f-65e483cf5494 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:51.619411 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc365fee-37db-4b16-b62f-65e483cf5494 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:51.637181 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-bc365fee-37db-4b16-b62f-65e483cf5494 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:51.637181 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:51.637181 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:51.637181 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:51.637181 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:51.637181 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:51.637181 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:51.637181 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:51.637181 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:51.637181 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:51.637181 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:51.637181 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:51.637181 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:33:51.637181 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:33:51.637181 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:33:51.637181 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:33:51.638231 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:51.638231 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:51.638231 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:51.638231 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:51.638231 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:51.638231 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:51.638231 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:51.638231 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:51.638231 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:51.638231 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:51.638231 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:51.638231 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:51.638231 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:51.638231 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:51.638231 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:51.638231 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:51.638231 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:33:51.639594 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:33:51.639594 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:51.639594 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:51.639594 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 254/524] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:33:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:51.640424 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc365fee-37db-4b16-b62f-65e483cf5494 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:51.691848 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ed0ae8-0a35-47e4-ac20-dd2e1c031617 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:51.728229 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e32db3e-0d58-4698-8671-56e8a375b397 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:51.728867 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e32db3e-0d58-4698-8671-56e8a375b397 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:51.729263 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e32db3e-0d58-4698-8671-56e8a375b397 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:51.729542 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e32db3e-0d58-4698-8671-56e8a375b397 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:51.738065 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e32db3e-0d58-4698-8671-56e8a375b397 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:51.739541 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e32db3e-0d58-4698-8671-56e8a375b397 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:51.746040 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:51.753781 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-3e32db3e-0d58-4698-8671-56e8a375b397 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:33:51.753781 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:51.753781 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:51.753781 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:51.753781 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:51.753781 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:51.753781 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:51.753781 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:51.753781 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:51.753781 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:51.753781 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:51.753781 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:51.753781 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:33:51.753781 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:33:51.753781 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:33:51.754774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:33:51.754774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:51.754774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:51.754774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:51.754774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:51.754774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:51.754774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:51.754774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:51.754774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:51.754774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:51.754774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:51.754774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:51.754774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:51.754774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:51.754774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:51.754774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:51.754774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:51.755612 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:33:51.755612 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:33:51.755612 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:33:51.755612 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:33:51.755612 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:33:51.755612 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:33:51.755612 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:33:51.755612 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:51.756057 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc365fee-37db-4b16-b62f-65e483cf5494 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:51.756125 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 271/525] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:33:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:51.756125 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e32db3e-0d58-4698-8671-56e8a375b397 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:51.788986 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:51.800734 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad30eb5-a1b9-4bd1-8ec4-216b21f565c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:51.801192 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad30eb5-a1b9-4bd1-8ec4-216b21f565c6 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:51.801473 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad30eb5-a1b9-4bd1-8ec4-216b21f565c6 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:51.801745 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad30eb5-a1b9-4bd1-8ec4-216b21f565c6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:51.808846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad30eb5-a1b9-4bd1-8ec4-216b21f565c6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:51.810302 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad30eb5-a1b9-4bd1-8ec4-216b21f565c6 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:51.823252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-fad30eb5-a1b9-4bd1-8ec4-216b21f565c6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:33:51.823252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:51.823252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:51.823252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:51.823252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:51.823252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:51.823252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:51.823252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:51.823252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:51.823252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:51.823252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:51.823252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:51.823252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:33:51.823252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:33:51.823252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:33:51.824186 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:33:51.824186 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:51.824186 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:51.824186 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:51.824186 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:51.824186 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:51.824186 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:51.824186 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:51.824186 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:51.824186 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:51.824186 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:51.824186 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:51.824186 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:51.824186 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:51.824186 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:51.824186 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:51.824186 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:51.825119 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:33:51.825119 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:33:51.825119 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:33:51.825119 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:33:51.825119 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:33:51.825119 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:33:51.825119 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:33:51.825119 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:51.825119 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 255/526] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:33:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:51.825660 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad30eb5-a1b9-4bd1-8ec4-216b21f565c6 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:51.884019 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e32db3e-0d58-4698-8671-56e8a375b397 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:51.925616 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a63559-3bd5-4768-8cb8-2917ff3bdd79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:51.926152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a63559-3bd5-4768-8cb8-2917ff3bdd79 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:51.926436 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a63559-3bd5-4768-8cb8-2917ff3bdd79 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:51.926720 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a63559-3bd5-4768-8cb8-2917ff3bdd79 None admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:51.927467 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a63559-3bd5-4768-8cb8-2917ff3bdd79 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:51.927821 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a63559-3bd5-4768-8cb8-2917ff3bdd79 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:51.929786 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a63559-3bd5-4768-8cb8-2917ff3bdd79 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:51.943857 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad30eb5-a1b9-4bd1-8ec4-216b21f565c6 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:51.951968 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 272/527] 10.222.0.102 () {56 vars in 1144 bytes} [Wed Aug 24 15:33:51 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 376 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:51.952599 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a63559-3bd5-4768-8cb8-2917ff3bdd79 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:51.988687 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37727a14-c027-499e-838e-1bd7ee424539 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:51.989286 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37727a14-c027-499e-838e-1bd7ee424539 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:51.989567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37727a14-c027-499e-838e-1bd7ee424539 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:51.989841 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37727a14-c027-499e-838e-1bd7ee424539 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:51.990533 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37727a14-c027-499e-838e-1bd7ee424539 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:51.990815 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37727a14-c027-499e-838e-1bd7ee424539 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:51.992245 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37727a14-c027-499e-838e-1bd7ee424539 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:52.005682 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 256/528] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:33:51 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:52.006644 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37727a14-c027-499e-838e-1bd7ee424539 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:52.025898 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:52.075985 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:52.095712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a63559-3bd5-4768-8cb8-2917ff3bdd79 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:52.125960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37727a14-c027-499e-838e-1bd7ee424539 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:52.135209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea1ab4a-a18c-4ff2-ba8b-ce6d398f7a18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:52.135846 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea1ab4a-a18c-4ff2-ba8b-ce6d398f7a18 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:52.136129 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea1ab4a-a18c-4ff2-ba8b-ce6d398f7a18 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:52.136525 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea1ab4a-a18c-4ff2-ba8b-ce6d398f7a18 None admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:52.137227 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea1ab4a-a18c-4ff2-ba8b-ce6d398f7a18 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:52.137537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea1ab4a-a18c-4ff2-ba8b-ce6d398f7a18 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:52.139681 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea1ab4a-a18c-4ff2-ba8b-ce6d398f7a18 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:52.164925 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 273/529] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:33:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:52.164925 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea1ab4a-a18c-4ff2-ba8b-ce6d398f7a18 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:52.170479 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7084247d-54eb-4f65-b4e1-a1658a5b6354 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:52.172002 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7084247d-54eb-4f65-b4e1-a1658a5b6354 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:52.172183 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7084247d-54eb-4f65-b4e1-a1658a5b6354 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:52.172417 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7084247d-54eb-4f65-b4e1-a1658a5b6354 None admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:52.173162 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7084247d-54eb-4f65-b4e1-a1658a5b6354 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:52.173480 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7084247d-54eb-4f65-b4e1-a1658a5b6354 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:52.174936 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7084247d-54eb-4f65-b4e1-a1658a5b6354 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:52.195340 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 257/530] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:33:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:52.196388 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7084247d-54eb-4f65-b4e1-a1658a5b6354 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:52.276114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea1ab4a-a18c-4ff2-ba8b-ce6d398f7a18 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:52.281840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:52.310034 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7084247d-54eb-4f65-b4e1-a1658a5b6354 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:52.314659 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7fb37d-5bd8-4ea4-987a-44ba612bf72e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:52.315664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7fb37d-5bd8-4ea4-987a-44ba612bf72e None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:52.316164 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7fb37d-5bd8-4ea4-987a-44ba612bf72e None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:52.316636 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7fb37d-5bd8-4ea4-987a-44ba612bf72e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:52.339891 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:52.352529 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a974e1-375a-44db-b8bb-05cc5ae7824d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:52.353168 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a974e1-375a-44db-b8bb-05cc5ae7824d None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:52.353450 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a974e1-375a-44db-b8bb-05cc5ae7824d None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:52.353759 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a974e1-375a-44db-b8bb-05cc5ae7824d None admin] PATH_INFO: `/v3/projects/service` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:52.361774 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a974e1-375a-44db-b8bb-05cc5ae7824d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:52.363244 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a974e1-375a-44db-b8bb-05cc5ae7824d None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:52.375299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-b1a974e1-375a-44db-b8bb-05cc5ae7824d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:33:52.375299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:52.375299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:52.375299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:52.375299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:52.375299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:52.375299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:52.375299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:52.375299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:52.375299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:52.375299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:52.375299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:52.375299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:33:52.375299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:33:52.375299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:33:52.376366 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:33:52.376366 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:52.376366 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:52.376366 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:52.376366 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:52.376366 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:52.376366 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:52.376366 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:52.376366 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:52.376366 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:52.376366 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:52.376366 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:52.376366 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:52.376366 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:52.376366 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:52.376366 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:52.376366 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:52.377213 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:33:52.377213 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:33:52.377213 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:33:52.377213 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:33:52.377213 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:33:52.377213 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:33:52.377213 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:33:52.377213 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:52.377213 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 258/531] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:33:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:52.377213 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a974e1-375a-44db-b8bb-05cc5ae7824d None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:52.410879 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c7fb37d-5bd8-4ea4-987a-44ba612bf72e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:52.412469 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c7fb37d-5bd8-4ea4-987a-44ba612bf72e None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:52.495994 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a974e1-375a-44db-b8bb-05cc5ae7824d None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:52.526453 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 274/532] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:33:52 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:33:52.527304 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7fb37d-5bd8-4ea4-987a-44ba612bf72e None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:52.531148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c0dc337-d8fb-4c49-bb52-a015735895e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:52.531783 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0dc337-d8fb-4c49-bb52-a015735895e9 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:52.532069 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0dc337-d8fb-4c49-bb52-a015735895e9 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:52.532342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0dc337-d8fb-4c49-bb52-a015735895e9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:52.533085 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c0dc337-d8fb-4c49-bb52-a015735895e9 None admin] RBAC: Adding query filter params (role.id=628bab09d1dc48fd9e01023b2f7671f6, scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5, user.id=8c6f06186c1a4626997db5a40c31cbb9) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:52.533400 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c0dc337-d8fb-4c49-bb52-a015735895e9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:52.534831 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c0dc337-d8fb-4c49-bb52-a015735895e9 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:52.536818 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:52.541811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 259/533] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:33:52 2022] GET /identity/v3/role_assignments?user.id=8c6f06186c1a4626997db5a40c31cbb9&scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5&role.id=628bab09d1dc48fd9e01023b2f7671f6 => generated 260 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:52.542595 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c0dc337-d8fb-4c49-bb52-a015735895e9 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:52.583535 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:52.661171 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7fb37d-5bd8-4ea4-987a-44ba612bf72e None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:52.675501 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c0dc337-d8fb-4c49-bb52-a015735895e9 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:52.704995 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6f2ce8-3be1-4594-a9fa-72825b8774af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:52.705585 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6f2ce8-3be1-4594-a9fa-72825b8774af None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:52.705873 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6f2ce8-3be1-4594-a9fa-72825b8774af None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:52.706782 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6f2ce8-3be1-4594-a9fa-72825b8774af None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:52.706888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6f2ce8-3be1-4594-a9fa-72825b8774af None admin] RBAC: Adding query filter params (role.id=628bab09d1dc48fd9e01023b2f7671f6, scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5, user.id=e0d544379fe44438ad10289b45124374) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:52.707256 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6f2ce8-3be1-4594-a9fa-72825b8774af None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:52.708723 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6f2ce8-3be1-4594-a9fa-72825b8774af None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:52.716396 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24406e10-baab-44b5-bbdd-eb1838921ab2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:52.717199 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24406e10-baab-44b5-bbdd-eb1838921ab2 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:52.717588 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24406e10-baab-44b5-bbdd-eb1838921ab2 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:52.717771 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24406e10-baab-44b5-bbdd-eb1838921ab2 None admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:52.718655 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24406e10-baab-44b5-bbdd-eb1838921ab2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:52.718755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 275/534] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:33:52 2022] GET /identity/v3/role_assignments?user.id=e0d544379fe44438ad10289b45124374&scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5&role.id=628bab09d1dc48fd9e01023b2f7671f6 => generated 260 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:52.718936 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24406e10-baab-44b5-bbdd-eb1838921ab2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:52.720335 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3001fd6-b736-4bfb-a838-de5801ed2e60 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:52.720335 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3001fd6-b736-4bfb-a838-de5801ed2e60 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:52.720335 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3001fd6-b736-4bfb-a838-de5801ed2e60 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:52.720575 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24406e10-baab-44b5-bbdd-eb1838921ab2 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:52.725857 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 276/535] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:52 2022] GET /identity => generated 271 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:52.726814 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3001fd6-b736-4bfb-a838-de5801ed2e60 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:52.735784 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 260/536] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:33:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:52.737246 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75dab46d-946e-4695-959d-74d7ba410044 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:52.737545 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75dab46d-946e-4695-959d-74d7ba410044 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:52.737817 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75dab46d-946e-4695-959d-74d7ba410044 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:52.758964 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-75dab46d-946e-4695-959d-74d7ba410044 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:52.760349 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-75dab46d-946e-4695-959d-74d7ba410044 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:52.831281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:52.861457 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 261/537] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:52 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:52.862621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75dab46d-946e-4695-959d-74d7ba410044 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:52.864568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3001fd6-b736-4bfb-a838-de5801ed2e60 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:52.891705 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:52.938091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa88ac28-3428-4d93-8de5-581903c883a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:52.938829 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa88ac28-3428-4d93-8de5-581903c883a9 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:52.939110 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa88ac28-3428-4d93-8de5-581903c883a9 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:52.939379 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa88ac28-3428-4d93-8de5-581903c883a9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:52.965894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa88ac28-3428-4d93-8de5-581903c883a9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:52.967716 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa88ac28-3428-4d93-8de5-581903c883a9 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:52.995755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-fa88ac28-3428-4d93-8de5-581903c883a9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:52.995755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:52.995755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:52.995755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:52.995755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:52.995755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:52.995755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:52.995755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:52.995755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:52.995755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:52.995755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:52.995755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:52.995755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:33:52.995755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:33:52.995755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:33:52.995755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:33:52.996700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:52.996700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:52.996700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:52.996700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:52.996700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:52.996700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:52.996700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:52.996700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:52.996700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:52.996700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:52.996700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:52.996700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:52.996700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:52.996700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:52.996700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:52.996700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:52.996700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:33:52.997608 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:33:52.997608 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:52.997608 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:52.997608 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 277/538] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:33:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:53.004333 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa88ac28-3428-4d93-8de5-581903c883a9 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:53.021933 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75dab46d-946e-4695-959d-74d7ba410044 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:53.122487 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385fad46-f5b0-46ac-a799-fbc7183afffe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:53.125060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385fad46-f5b0-46ac-a799-fbc7183afffe None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:53.126100 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385fad46-f5b0-46ac-a799-fbc7183afffe None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:53.127240 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385fad46-f5b0-46ac-a799-fbc7183afffe None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:53.128390 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-385fad46-f5b0-46ac-a799-fbc7183afffe None admin] RBAC: Adding query filter params (role.id=628bab09d1dc48fd9e01023b2f7671f6, scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5, user.id=08fbd28d9593476b8753f8d64ee64f5b) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:53.130016 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-385fad46-f5b0-46ac-a799-fbc7183afffe None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:53.131729 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-385fad46-f5b0-46ac-a799-fbc7183afffe None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:53.137759 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa88ac28-3428-4d93-8de5-581903c883a9 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:53.138841 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 262/539] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:33:52 2022] GET /identity/v3/role_assignments?user.id=08fbd28d9593476b8753f8d64ee64f5b&scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5&role.id=628bab09d1dc48fd9e01023b2f7671f6 => generated 260 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:53.186408 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8038c98-1aec-495f-a5e9-bdedc16b2092 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:53.187813 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8038c98-1aec-495f-a5e9-bdedc16b2092 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:53.188359 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8038c98-1aec-495f-a5e9-bdedc16b2092 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:53.188822 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8038c98-1aec-495f-a5e9-bdedc16b2092 None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:53.189699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8038c98-1aec-495f-a5e9-bdedc16b2092 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:53.190219 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8038c98-1aec-495f-a5e9-bdedc16b2092 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:53.191850 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8038c98-1aec-495f-a5e9-bdedc16b2092 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:53.214059 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 278/540] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:33:53 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:53.220811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385fad46-f5b0-46ac-a799-fbc7183afffe None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:53.267186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8038c98-1aec-495f-a5e9-bdedc16b2092 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:53.302735 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:53.319162 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:53.368088 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385fad46-f5b0-46ac-a799-fbc7183afffe None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:53.407091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8038c98-1aec-495f-a5e9-bdedc16b2092 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:53.408431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb62c2c-14f9-4db9-af47-0a67c28fac6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:53.409342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb62c2c-14f9-4db9-af47-0a67c28fac6f None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:53.409856 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb62c2c-14f9-4db9-af47-0a67c28fac6f None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:53.410342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb62c2c-14f9-4db9-af47-0a67c28fac6f None admin] PATH_INFO: `/v3/projects/service` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:53.418171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcb62c2c-14f9-4db9-af47-0a67c28fac6f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:53.420089 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcb62c2c-14f9-4db9-af47-0a67c28fac6f None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:53.432995 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-fcb62c2c-14f9-4db9-af47-0a67c28fac6f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:33:53.432995 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:53.432995 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:53.432995 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:53.432995 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:53.432995 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:53.432995 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:53.432995 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:53.432995 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:53.432995 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:53.432995 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:53.432995 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:53.432995 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:33:53.432995 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:33:53.432995 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:33:53.433912 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:33:53.433912 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:53.433912 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:53.433912 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:53.433912 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:53.433912 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:53.433912 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:53.433912 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:53.433912 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:53.433912 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:53.433912 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:53.433912 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:53.433912 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:53.433912 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:53.433912 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:53.433912 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:53.433912 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:53.434751 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:33:53.434751 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:33:53.434751 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:33:53.434751 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:33:53.434751 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:33:53.434751 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:33:53.434751 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:33:53.434751 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:53.438014 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 263/541] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:33:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:53.444124 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb62c2c-14f9-4db9-af47-0a67c28fac6f None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:53.446996 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d3eead-8b2f-4851-a3b0-1c16e36165cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:53.447689 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d3eead-8b2f-4851-a3b0-1c16e36165cc None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:53.448044 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d3eead-8b2f-4851-a3b0-1c16e36165cc None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:53.448228 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d3eead-8b2f-4851-a3b0-1c16e36165cc None admin] PATH_INFO: `/v3/roles/service` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:53.459929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21d3eead-8b2f-4851-a3b0-1c16e36165cc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:53.464826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-21d3eead-8b2f-4851-a3b0-1c16e36165cc None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:33:53.464826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:53.464826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:53.464826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:53.464826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:53.464826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:53.464826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:53.464826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:53.464826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:53.464826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:53.464826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:53.464826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:53.464826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:53.464826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:53.464826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:53.464826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:53.465713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:53.465713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:53.465713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:53.465713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:53.465713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:53.465713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:53.465713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:53.465713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:53.465713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:53.465713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:53.465713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:53.465713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:53.465713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:53.465713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:53.465713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:53.465713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:53.466485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:53.466485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:53.466485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:53.466485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:53.466485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:53.466485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:53.466485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:53.466485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:53.466485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:53.466485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:53.466485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:33:53.466485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:53.466485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 279/542] 10.222.0.102 () {56 vars in 1107 bytes} [Wed Aug 24 15:33:53 2022] GET /identity/v3/roles/service => generated 85 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:33:53.472216 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d3eead-8b2f-4851-a3b0-1c16e36165cc None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:53.559736 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:53.566098 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb62c2c-14f9-4db9-af47-0a67c28fac6f None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:53.580651 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d3eead-8b2f-4851-a3b0-1c16e36165cc None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:53.581913 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:53.626151 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb97826-0cb8-4fe1-af59-156d3aa9c203 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:53.626760 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb97826-0cb8-4fe1-af59-156d3aa9c203 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:53.627044 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb97826-0cb8-4fe1-af59-156d3aa9c203 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:53.627312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb97826-0cb8-4fe1-af59-156d3aa9c203 None admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:53.628032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb97826-0cb8-4fe1-af59-156d3aa9c203 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:53.628338 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb97826-0cb8-4fe1-af59-156d3aa9c203 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:53.629815 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb97826-0cb8-4fe1-af59-156d3aa9c203 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:53.632749 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362c1a69-649d-4a88-a63d-9137b9926be1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:53.633413 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362c1a69-649d-4a88-a63d-9137b9926be1 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:53.633692 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362c1a69-649d-4a88-a63d-9137b9926be1 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:53.633960 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362c1a69-649d-4a88-a63d-9137b9926be1 None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:53.634651 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362c1a69-649d-4a88-a63d-9137b9926be1 None admin] RBAC: Adding query filter params (name=service) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:53.634980 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362c1a69-649d-4a88-a63d-9137b9926be1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:53.636405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362c1a69-649d-4a88-a63d-9137b9926be1 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:53.653823 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 264/543] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:33:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:53.658624 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 280/544] 10.222.0.102 () {56 vars in 1108 bytes} [Wed Aug 24 15:33:53 2022] GET /identity/v3/roles?name=service => generated 328 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:33:53.663128 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb97826-0cb8-4fe1-af59-156d3aa9c203 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:53.668644 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362c1a69-649d-4a88-a63d-9137b9926be1 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:53.831234 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362c1a69-649d-4a88-a63d-9137b9926be1 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:53.868285 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a7161c-839c-43ed-8e1d-f514b66070c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:53.868917 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a7161c-839c-43ed-8e1d-f514b66070c4 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:53.869195 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a7161c-839c-43ed-8e1d-f514b66070c4 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:53.869466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a7161c-839c-43ed-8e1d-f514b66070c4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:53.881530 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0a7161c-839c-43ed-8e1d-f514b66070c4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:53.883089 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0a7161c-839c-43ed-8e1d-f514b66070c4 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:53.885299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb97826-0cb8-4fe1-af59-156d3aa9c203 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:53.885908 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:53.901664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-f0a7161c-839c-43ed-8e1d-f514b66070c4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:53.901664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:53.901664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:53.901664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:53.901664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:53.901664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:53.901664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:53.901664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:53.901664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:53.901664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:53.901664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:53.901664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:53.901664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:33:53.901664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:33:53.901664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:33:53.901664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:33:53.902633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:53.902633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:53.902633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:53.902633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:53.902633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:53.902633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:53.902633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:53.902633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:53.902633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:53.902633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:53.902633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:53.902633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:53.902633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:53.902633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:53.902633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:53.902633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:53.902633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:33:53.903445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:33:53.903445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:53.903445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:53.906402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 281/545] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:33:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:53.912995 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a7161c-839c-43ed-8e1d-f514b66070c4 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:53.962348 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14494847-795e-4843-b84a-65e2e5cbecf9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:53.963375 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14494847-795e-4843-b84a-65e2e5cbecf9 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:53.963868 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14494847-795e-4843-b84a-65e2e5cbecf9 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:53.965195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14494847-795e-4843-b84a-65e2e5cbecf9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:53.967765 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14494847-795e-4843-b84a-65e2e5cbecf9 None admin] RBAC: Adding query filter params (role.id=628bab09d1dc48fd9e01023b2f7671f6, scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5, user.id=48939674c1fd4cf599691d510fcd8858) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:53.969640 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14494847-795e-4843-b84a-65e2e5cbecf9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:53.971355 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14494847-795e-4843-b84a-65e2e5cbecf9 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:53.984983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 265/546] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:33:53 2022] GET /identity/v3/role_assignments?user.id=48939674c1fd4cf599691d510fcd8858&scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5&role.id=628bab09d1dc48fd9e01023b2f7671f6 => generated 260 bytes in 322 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:54.067854 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a7161c-839c-43ed-8e1d-f514b66070c4 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:54.109003 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e1cef95-33e0-4ef5-86cd-aee45be2930e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:54.109961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1cef95-33e0-4ef5-86cd-aee45be2930e None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:54.110430 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1cef95-33e0-4ef5-86cd-aee45be2930e None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:54.110900 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1cef95-33e0-4ef5-86cd-aee45be2930e None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:54.111789 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e1cef95-33e0-4ef5-86cd-aee45be2930e None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:54.112300 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e1cef95-33e0-4ef5-86cd-aee45be2930e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:54.113858 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e1cef95-33e0-4ef5-86cd-aee45be2930e None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:54.128226 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 282/547] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:33:53 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:33:54.134314 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14494847-795e-4843-b84a-65e2e5cbecf9 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:54.177313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:54.254569 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14494847-795e-4843-b84a-65e2e5cbecf9 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:54.303238 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcadb2c4-077e-4a62-8a0c-cb99132d4054 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:54.304232 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcadb2c4-077e-4a62-8a0c-cb99132d4054 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:54.304706 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcadb2c4-077e-4a62-8a0c-cb99132d4054 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:54.305272 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcadb2c4-077e-4a62-8a0c-cb99132d4054 None admin] PATH_INFO: `/v3/users/swift` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:54.315720 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcadb2c4-077e-4a62-8a0c-cb99132d4054 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:54.317467 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcadb2c4-077e-4a62-8a0c-cb99132d4054 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:54.329357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-dcadb2c4-077e-4a62-8a0c-cb99132d4054 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 24 15:33:54.329357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:54.329357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:54.329357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:54.329357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:54.329357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:54.329357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:54.329357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:54.329357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:54.329357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:54.329357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:54.329357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:54.329357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:33:54.329357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:33:54.329357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:33:54.329357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:33:54.330291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:54.330291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:54.330291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:33:54.330291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:33:54.330291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:33:54.330291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:33:54.330291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:54.330291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:54.330291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:54.330291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:54.330291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:54.330291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:54.330291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:54.330291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:54.330291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:54.330291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:54.330291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:54.330291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:54.331773 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:54.331773 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:54.331773 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:33:54.331773 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:33:54.331773 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:33:54.331773 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:33:54.331773 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:33:54.331773 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:33:54.331773 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 24 15:33:54.331773 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:54.334371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 266/548] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:33:54 2022] GET /identity/v3/users/swift => generated 83 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:54.340283 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e1cef95-33e0-4ef5-86cd-aee45be2930e None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:54.405963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:54.454900 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e1cef95-33e0-4ef5-86cd-aee45be2930e None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:54.492660 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8922bb7c-473d-4904-8d26-29696f9f4e6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:54.493817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8922bb7c-473d-4904-8d26-29696f9f4e6f None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:54.494291 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8922bb7c-473d-4904-8d26-29696f9f4e6f None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:54.494809 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8922bb7c-473d-4904-8d26-29696f9f4e6f None admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:54.495762 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8922bb7c-473d-4904-8d26-29696f9f4e6f None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:54.496262 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8922bb7c-473d-4904-8d26-29696f9f4e6f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:54.497909 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8922bb7c-473d-4904-8d26-29696f9f4e6f None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:54.518244 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 283/549] 10.222.0.102 () {56 vars in 1140 bytes} [Wed Aug 24 15:33:54 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 372 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:54.524098 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcadb2c4-077e-4a62-8a0c-cb99132d4054 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:54.649414 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:54.660593 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcadb2c4-077e-4a62-8a0c-cb99132d4054 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:54.702454 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4600bf-5ef9-484f-b268-bc5a83aa90c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:54.703452 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4600bf-5ef9-484f-b268-bc5a83aa90c9 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:54.703998 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4600bf-5ef9-484f-b268-bc5a83aa90c9 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:54.704466 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4600bf-5ef9-484f-b268-bc5a83aa90c9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:54.750853 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9dc46c-e35d-4096-b11c-168472dc2c31 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:54.751404 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9dc46c-e35d-4096-b11c-168472dc2c31 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:54.751624 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9dc46c-e35d-4096-b11c-168472dc2c31 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:54.753277 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 284/550] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:54 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:33:54.758857 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9be3751-f658-4a90-a5c5-09c9ddede0d5 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:54.759327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9be3751-f658-4a90-a5c5-09c9ddede0d5 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:54.759808 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9be3751-f658-4a90-a5c5-09c9ddede0d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:54.787505 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-a9be3751-f658-4a90-a5c5-09c9ddede0d5 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:54.788859 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-a9be3751-f658-4a90-a5c5-09c9ddede0d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:54.814184 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4600bf-5ef9-484f-b268-bc5a83aa90c9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:54.815973 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4600bf-5ef9-484f-b268-bc5a83aa90c9 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:54.883659 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 285/551] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:54 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 15:33:54.991805 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:55.002313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 267/552] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:33:54 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 479 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:33:55.007893 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9be3751-f658-4a90-a5c5-09c9ddede0d5 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:55.061954 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:55.124081 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9be3751-f658-4a90-a5c5-09c9ddede0d5 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:55.161678 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5219b5de-713c-4d77-90ed-d44e9ef65e9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:55.162321 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5219b5de-713c-4d77-90ed-d44e9ef65e9b None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:55.162603 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5219b5de-713c-4d77-90ed-d44e9ef65e9b None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:55.162877 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5219b5de-713c-4d77-90ed-d44e9ef65e9b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:55.174944 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5219b5de-713c-4d77-90ed-d44e9ef65e9b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:55.176697 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5219b5de-713c-4d77-90ed-d44e9ef65e9b None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:55.194478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-5219b5de-713c-4d77-90ed-d44e9ef65e9b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:55.194478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:55.194478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:55.194478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:55.194478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:55.194478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:55.194478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:55.194478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:55.194478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:55.194478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:55.194478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:55.194478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:55.194478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:33:55.194478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:33:55.194478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:33:55.194478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:33:55.195557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:55.195557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:55.195557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:55.195557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:55.195557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:55.195557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:55.195557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:55.195557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:55.195557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:55.195557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:55.195557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:55.195557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:55.195557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:55.195557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:55.195557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:55.195557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:55.195557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:33:55.196447 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:33:55.196447 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:55.196447 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:55.198995 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 286/553] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:33:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:55.202135 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4600bf-5ef9-484f-b268-bc5a83aa90c9 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:55.249738 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5219b5de-713c-4d77-90ed-d44e9ef65e9b None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:55.363334 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:55.371169 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5219b5de-713c-4d77-90ed-d44e9ef65e9b None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:55.400815 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4600bf-5ef9-484f-b268-bc5a83aa90c9 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:55.414090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec40d6a5-6ad1-481a-bdc4-0abb9aa1995c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:55.414854 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec40d6a5-6ad1-481a-bdc4-0abb9aa1995c None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:55.415143 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec40d6a5-6ad1-481a-bdc4-0abb9aa1995c None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:55.415420 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec40d6a5-6ad1-481a-bdc4-0abb9aa1995c None admin] PATH_INFO: `/v3/roles/service` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:55.428155 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec40d6a5-6ad1-481a-bdc4-0abb9aa1995c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:55.433878 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-ec40d6a5-6ad1-481a-bdc4-0abb9aa1995c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:33:55.433878 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:55.433878 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:55.433878 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:55.433878 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:55.433878 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:55.433878 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:55.433878 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:55.433878 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:55.433878 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:55.433878 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:55.433878 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:55.433878 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:55.433878 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:55.433878 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:55.433878 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:55.434769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:55.434769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:55.434769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:55.434769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:55.434769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:55.434769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:55.434769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:55.434769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:55.434769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:55.434769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:55.434769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:55.434769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:55.434769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:55.434769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:55.434769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:55.434769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:55.435935 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:55.435935 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:55.435935 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:55.435935 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:55.435935 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:55.435935 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:55.435935 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:55.435935 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:55.435935 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:55.435935 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:55.435935 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:33:55.435935 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:55.435935 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 287/554] 10.222.0.102 () {56 vars in 1107 bytes} [Wed Aug 24 15:33:55 2022] GET /identity/v3/roles/service => generated 85 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:55.436664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db1f02e-cee0-4d3b-830b-06727d67ba54 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:55.437044 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db1f02e-cee0-4d3b-830b-06727d67ba54 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:55.437209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db1f02e-cee0-4d3b-830b-06727d67ba54 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:55.442736 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 288/555] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:55 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:55.442736 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db1f02e-cee0-4d3b-830b-06727d67ba54 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:55.454255 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b83857e-a69f-4ad3-9426-d5e491965026 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:55.454871 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b83857e-a69f-4ad3-9426-d5e491965026 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:55.455157 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b83857e-a69f-4ad3-9426-d5e491965026 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:55.455436 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b83857e-a69f-4ad3-9426-d5e491965026 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:55.456128 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b83857e-a69f-4ad3-9426-d5e491965026 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:55.456440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b83857e-a69f-4ad3-9426-d5e491965026 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:55.458203 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b83857e-a69f-4ad3-9426-d5e491965026 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:55.473653 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 268/556] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:33:55 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:33:55.476671 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231f90f4-5048-484b-985c-bd98dbb61b7d None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:55.477335 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231f90f4-5048-484b-985c-bd98dbb61b7d None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:55.477821 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231f90f4-5048-484b-985c-bd98dbb61b7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:55.498916 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-231f90f4-5048-484b-985c-bd98dbb61b7d None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:55.500710 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-231f90f4-5048-484b-985c-bd98dbb61b7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:55.557603 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:55.583657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db1f02e-cee0-4d3b-830b-06727d67ba54 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:55.584029 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 269/557] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:55 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:55.585147 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231f90f4-5048-484b-985c-bd98dbb61b7d None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:55.619656 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e13a52-64f3-48f8-b8ff-621829a63b26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:55.619656 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e13a52-64f3-48f8-b8ff-621829a63b26 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:55.619656 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e13a52-64f3-48f8-b8ff-621829a63b26 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:55.619656 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e13a52-64f3-48f8-b8ff-621829a63b26 None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:55.620057 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e13a52-64f3-48f8-b8ff-621829a63b26 None admin] RBAC: Adding query filter params (name=service) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:55.620057 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e13a52-64f3-48f8-b8ff-621829a63b26 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:55.620574 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e13a52-64f3-48f8-b8ff-621829a63b26 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:55.635782 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 289/558] 10.222.0.102 () {56 vars in 1108 bytes} [Wed Aug 24 15:33:55 2022] GET /identity/v3/roles?name=service => generated 328 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:55.636319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e13a52-64f3-48f8-b8ff-621829a63b26 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:55.678520 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:55.791320 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231f90f4-5048-484b-985c-bd98dbb61b7d None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:55.818529 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e13a52-64f3-48f8-b8ff-621829a63b26 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:55.840437 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cbfdcdd-e18e-40e4-82d2-c09ec47b2a20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:55.845004 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbfdcdd-e18e-40e4-82d2-c09ec47b2a20 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:55.845004 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbfdcdd-e18e-40e4-82d2-c09ec47b2a20 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:55.845004 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbfdcdd-e18e-40e4-82d2-c09ec47b2a20 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:55.856257 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cbfdcdd-e18e-40e4-82d2-c09ec47b2a20 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:55.856653 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cbfdcdd-e18e-40e4-82d2-c09ec47b2a20 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:55.858491 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e9d644-7152-42e5-ab8d-f12d66aa48a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:55.860084 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e9d644-7152-42e5-ab8d-f12d66aa48a6 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:55.860371 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e9d644-7152-42e5-ab8d-f12d66aa48a6 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:55.860653 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e9d644-7152-42e5-ab8d-f12d66aa48a6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:55.871386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-8cbfdcdd-e18e-40e4-82d2-c09ec47b2a20 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:33:55.871386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:55.871386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:55.871386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:55.871386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:55.871386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:55.871386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:55.871386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:55.871386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:55.871386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:55.871386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:55.871386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:55.871386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:33:55.871386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:33:55.871386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:33:55.872341 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:33:55.872341 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:55.872341 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:55.872341 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:55.872341 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:55.872341 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:55.872341 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:55.872341 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:55.872341 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:55.872341 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:55.872341 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:55.872341 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:55.872341 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:55.872341 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:55.872341 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:55.872341 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:55.872341 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:55.873291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:33:55.873291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:33:55.873291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:33:55.873291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:33:55.873291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:33:55.873291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:33:55.873291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:33:55.873291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:55.873716 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e9d644-7152-42e5-ab8d-f12d66aa48a6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:55.875831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e9d644-7152-42e5-ab8d-f12d66aa48a6 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:55.877901 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 270/559] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:33:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 288 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:55.879379 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba490fe0-7a7d-42ca-a78b-36ad7879a524 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:55.879972 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba490fe0-7a7d-42ca-a78b-36ad7879a524 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:55.880254 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba490fe0-7a7d-42ca-a78b-36ad7879a524 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:55.882341 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 271/560] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:55 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:55.883081 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba490fe0-7a7d-42ca-a78b-36ad7879a524 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:55.898704 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-c5e9d644-7152-42e5-ab8d-f12d66aa48a6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:55.898704 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:55.898704 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:55.898704 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:55.898704 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:55.898704 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:55.898704 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:55.898704 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:55.898704 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:55.898704 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:55.898704 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:55.898704 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:55.898704 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:33:55.898704 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:33:55.898704 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:33:55.898704 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:33:55.901030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:55.901030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:55.901030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:55.901030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:55.901030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:55.901030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:55.901030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:55.901030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:55.901030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:55.901030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:55.901030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:55.901030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:55.901030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:55.901030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:55.901030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:55.901030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:55.901030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:33:55.903152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:33:55.903152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:55.903152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:55.907676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 290/561] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:33:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 264 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:55.907676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f179ce0-cf2d-4e75-aede-e6745358b13d None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:55.907676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f179ce0-cf2d-4e75-aede-e6745358b13d None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:55.908012 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f179ce0-cf2d-4e75-aede-e6745358b13d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:55.928636 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-1f179ce0-cf2d-4e75-aede-e6745358b13d None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:55.930245 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-1f179ce0-cf2d-4e75-aede-e6745358b13d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:55.946299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:56.013047 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:56.019765 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 291/562] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:55 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:56.019925 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba490fe0-7a7d-42ca-a78b-36ad7879a524 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:56.021014 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f179ce0-cf2d-4e75-aede-e6745358b13d None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:56.068073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76ac3c5-23f0-46d2-b0df-f9b2f84ed55e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:56.068593 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76ac3c5-23f0-46d2-b0df-f9b2f84ed55e None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:56.068888 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76ac3c5-23f0-46d2-b0df-f9b2f84ed55e None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:56.069158 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76ac3c5-23f0-46d2-b0df-f9b2f84ed55e None admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:56.069861 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76ac3c5-23f0-46d2-b0df-f9b2f84ed55e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:56.070168 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76ac3c5-23f0-46d2-b0df-f9b2f84ed55e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:56.071829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76ac3c5-23f0-46d2-b0df-f9b2f84ed55e None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:56.090127 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 272/563] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:33:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:56.090127 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76ac3c5-23f0-46d2-b0df-f9b2f84ed55e None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:56.148856 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f179ce0-cf2d-4e75-aede-e6745358b13d None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:56.186670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9190c667-c46e-46ed-b509-a241c78096e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:56.187974 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9190c667-c46e-46ed-b509-a241c78096e9 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:56.188464 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9190c667-c46e-46ed-b509-a241c78096e9 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:56.188957 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9190c667-c46e-46ed-b509-a241c78096e9 None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:56.189810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9190c667-c46e-46ed-b509-a241c78096e9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:56.190314 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9190c667-c46e-46ed-b509-a241c78096e9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:56.192158 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9190c667-c46e-46ed-b509-a241c78096e9 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:56.207653 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 292/564] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:33:56 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:56.209104 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9190c667-c46e-46ed-b509-a241c78096e9 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:56.228736 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76ac3c5-23f0-46d2-b0df-f9b2f84ed55e None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:56.237255 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:56.269076 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9177d344-9feb-4b2b-98ae-616308fe4129 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:56.269417 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9177d344-9feb-4b2b-98ae-616308fe4129 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:56.269805 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9177d344-9feb-4b2b-98ae-616308fe4129 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:56.270181 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9177d344-9feb-4b2b-98ae-616308fe4129 None admin] PATH_INFO: `/v3/roles/service` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:56.284375 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9177d344-9feb-4b2b-98ae-616308fe4129 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:56.288886 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:56.291496 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-9177d344-9feb-4b2b-98ae-616308fe4129 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:33:56.291496 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:56.291496 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:56.291496 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:56.291496 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:56.291496 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:56.291496 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:56.291496 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:56.291496 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:56.291496 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:56.291496 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:56.291496 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:56.291496 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:56.291496 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:56.291496 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:56.291496 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:56.292550 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:56.292550 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:56.292550 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:56.292550 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:56.292550 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:56.292550 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:56.292550 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:56.292550 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:56.292550 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:56.292550 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:56.292550 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:56.292550 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:56.292550 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:56.292550 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:56.292550 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:56.292550 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:56.293585 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:56.293585 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:56.293585 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:56.293585 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:56.293585 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:56.293585 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:56.293585 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:56.293585 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:56.293585 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:56.293585 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:56.293585 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:33:56.293585 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:56.296613 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 273/565] 10.222.0.102 () {56 vars in 1107 bytes} [Wed Aug 24 15:33:56 2022] GET /identity/v3/roles/service => generated 85 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:56.297591 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9177d344-9feb-4b2b-98ae-616308fe4129 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:56.348668 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9190c667-c46e-46ed-b509-a241c78096e9 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:56.399835 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac71c1bb-4c3b-4c3d-8e18-b9241344a636 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:56.400444 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac71c1bb-4c3b-4c3d-8e18-b9241344a636 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:56.400725 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac71c1bb-4c3b-4c3d-8e18-b9241344a636 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:56.400997 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac71c1bb-4c3b-4c3d-8e18-b9241344a636 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:56.401843 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac71c1bb-4c3b-4c3d-8e18-b9241344a636 None admin] RBAC: Adding query filter params (role.id=628bab09d1dc48fd9e01023b2f7671f6, scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5, user.id=6a070d4fd8c4466c857bd03965c11349) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:56.402247 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac71c1bb-4c3b-4c3d-8e18-b9241344a636 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:56.403964 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac71c1bb-4c3b-4c3d-8e18-b9241344a636 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:56.410941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 293/566] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:33:56 2022] GET /identity/v3/role_assignments?user.id=6a070d4fd8c4466c857bd03965c11349&scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5&role.id=628bab09d1dc48fd9e01023b2f7671f6 => generated 260 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:56.411584 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac71c1bb-4c3b-4c3d-8e18-b9241344a636 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:56.434473 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9177d344-9feb-4b2b-98ae-616308fe4129 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:56.477303 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b85a9e-e7b9-4855-abfa-a66b52add290 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:56.477951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b85a9e-e7b9-4855-abfa-a66b52add290 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:56.478312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b85a9e-e7b9-4855-abfa-a66b52add290 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:56.478582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b85a9e-e7b9-4855-abfa-a66b52add290 None admin] PATH_INFO: `/v3/users/glance` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:56.487191 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b85a9e-e7b9-4855-abfa-a66b52add290 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:56.488728 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b85a9e-e7b9-4855-abfa-a66b52add290 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:56.502342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-d3b85a9e-e7b9-4855-abfa-a66b52add290 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 24 15:33:56.502342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:56.502342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:56.502342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:56.502342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:56.502342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:56.502342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:56.502342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:56.502342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:56.502342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:56.502342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:56.502342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:56.502342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:33:56.502342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:33:56.502342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:33:56.502342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:33:56.503371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:56.503371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:56.503371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:33:56.503371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:33:56.503371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:33:56.503371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:33:56.503371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:56.503371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:56.503371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:56.503371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:56.503371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:56.503371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:56.503371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:56.503371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:56.503371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:56.503371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:56.503371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:56.503371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:56.504332 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:56.504332 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:56.504332 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:33:56.504332 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:33:56.504332 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:33:56.504332 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:33:56.504332 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:33:56.504332 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:33:56.504332 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 24 15:33:56.504332 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:56.504332 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 274/567] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:33:56 2022] GET /identity/v3/users/glance => generated 84 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:56.504332 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b85a9e-e7b9-4855-abfa-a66b52add290 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:56.540513 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:56.552291 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac71c1bb-4c3b-4c3d-8e18-b9241344a636 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:56.585040 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:56.602252 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c65df3-db29-4e93-911c-3ec4a6fa21fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:56.602781 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c65df3-db29-4e93-911c-3ec4a6fa21fd None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:56.603059 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c65df3-db29-4e93-911c-3ec4a6fa21fd None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:56.603328 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c65df3-db29-4e93-911c-3ec4a6fa21fd None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:56.609058 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c65df3-db29-4e93-911c-3ec4a6fa21fd None admin] RBAC: Adding query filter params (name=service) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:56.609446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c65df3-db29-4e93-911c-3ec4a6fa21fd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:56.611040 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c65df3-db29-4e93-911c-3ec4a6fa21fd None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:56.626509 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 294/568] 10.222.0.102 () {56 vars in 1108 bytes} [Wed Aug 24 15:33:56 2022] GET /identity/v3/roles?name=service => generated 328 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:56.626509 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36b69ed-0040-4745-b1a6-e45acba27ece None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:56.626509 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36b69ed-0040-4745-b1a6-e45acba27ece None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:56.626509 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36b69ed-0040-4745-b1a6-e45acba27ece None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:56.626509 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 295/569] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:56 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:56.626509 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36b69ed-0040-4745-b1a6-e45acba27ece None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:56.634270 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b85a9e-e7b9-4855-abfa-a66b52add290 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:56.672988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611cc093-8f4c-4c2f-9dc2-5acb8bf80dc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:56.673778 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611cc093-8f4c-4c2f-9dc2-5acb8bf80dc3 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:56.674066 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611cc093-8f4c-4c2f-9dc2-5acb8bf80dc3 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:56.674341 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611cc093-8f4c-4c2f-9dc2-5acb8bf80dc3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:56.688052 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-611cc093-8f4c-4c2f-9dc2-5acb8bf80dc3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:56.693489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-611cc093-8f4c-4c2f-9dc2-5acb8bf80dc3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:33:56.693489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:56.693489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:56.693489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:56.693489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:56.693489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:56.693489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:56.693489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:56.693489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:56.693489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:56.693489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:56.693489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:56.693489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:56.693489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:56.693489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:56.693489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:56.694544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:56.694544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:56.694544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:56.694544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:56.694544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:56.694544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:56.694544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:56.694544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:56.694544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:56.694544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:56.694544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:56.694544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:56.694544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:56.694544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:56.694544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:56.694544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:56.695592 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:56.695592 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:56.695592 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:56.695592 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:56.695592 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:56.695592 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:56.695592 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:56.695592 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:56.695592 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:56.695592 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:56.695592 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:33:56.695592 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:56.695592 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 275/570] 10.222.0.102 () {56 vars in 1107 bytes} [Wed Aug 24 15:33:56 2022] GET /identity/v3/roles/service => generated 85 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:56.695592 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611cc093-8f4c-4c2f-9dc2-5acb8bf80dc3 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:56.760498 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36b69ed-0040-4745-b1a6-e45acba27ece None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:56.800678 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b30603c-2409-4f8d-9bf5-5684d6e0ac7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:56.801346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b30603c-2409-4f8d-9bf5-5684d6e0ac7e None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:56.801631 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b30603c-2409-4f8d-9bf5-5684d6e0ac7e None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:56.801911 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b30603c-2409-4f8d-9bf5-5684d6e0ac7e None admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:56.802657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b30603c-2409-4f8d-9bf5-5684d6e0ac7e None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:56.802974 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b30603c-2409-4f8d-9bf5-5684d6e0ac7e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:56.804468 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b30603c-2409-4f8d-9bf5-5684d6e0ac7e None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:56.807357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:56.830253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 296/571] 10.222.0.102 () {56 vars in 1142 bytes} [Wed Aug 24 15:33:56 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 374 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:56.830253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5439a9cb-fbec-4957-967f-722efdc21033 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:56.830253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5439a9cb-fbec-4957-967f-722efdc21033 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:56.830253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5439a9cb-fbec-4957-967f-722efdc21033 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:56.839948 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611cc093-8f4c-4c2f-9dc2-5acb8bf80dc3 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:56.847054 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-5439a9cb-fbec-4957-967f-722efdc21033 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:56.848289 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-5439a9cb-fbec-4957-967f-722efdc21033 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:56.891776 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd1d73c-eaf8-41db-9ba8-b12a06f7dfe7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:56.892415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd1d73c-eaf8-41db-9ba8-b12a06f7dfe7 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:56.892692 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd1d73c-eaf8-41db-9ba8-b12a06f7dfe7 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:56.892961 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd1d73c-eaf8-41db-9ba8-b12a06f7dfe7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:56.894359 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:56.905684 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd1d73c-eaf8-41db-9ba8-b12a06f7dfe7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:56.907134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd1d73c-eaf8-41db-9ba8-b12a06f7dfe7 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:56.924148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-2fd1d73c-eaf8-41db-9ba8-b12a06f7dfe7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:56.924148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:56.924148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:56.924148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:56.924148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:56.924148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:56.924148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:56.924148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:56.924148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:56.924148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:56.924148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:56.924148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:56.924148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:33:56.924148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:33:56.924148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:33:56.924148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:33:56.925156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:56.925156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:56.925156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:56.925156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:56.925156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:56.925156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:56.925156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:56.925156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:56.925156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:56.925156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:56.925156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:56.925156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:56.925156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:56.925156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:56.925156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:56.925156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:56.925156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:33:56.926113 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:33:56.926113 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:56.926113 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:56.926113 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 276/572] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:33:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:56.926823 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd1d73c-eaf8-41db-9ba8-b12a06f7dfe7 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:56.931162 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 297/573] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:56 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:56.932613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5439a9cb-fbec-4957-967f-722efdc21033 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:57.060139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd1d73c-eaf8-41db-9ba8-b12a06f7dfe7 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:57.078708 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5439a9cb-fbec-4957-967f-722efdc21033 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:57.112353 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05705923-7f74-4483-897f-c37644b6a9b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:57.113114 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05705923-7f74-4483-897f-c37644b6a9b9 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:57.113511 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05705923-7f74-4483-897f-c37644b6a9b9 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:57.113899 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05705923-7f74-4483-897f-c37644b6a9b9 None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:57.114696 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05705923-7f74-4483-897f-c37644b6a9b9 None admin] RBAC: Adding query filter params (name=service) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:57.115119 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05705923-7f74-4483-897f-c37644b6a9b9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:57.116619 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05705923-7f74-4483-897f-c37644b6a9b9 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:57.122396 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a54d453-2ff3-4329-a379-e4222d4df957 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:57.122876 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:57.124128 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a54d453-2ff3-4329-a379-e4222d4df957 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:57.124674 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a54d453-2ff3-4329-a379-e4222d4df957 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:57.125101 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a54d453-2ff3-4329-a379-e4222d4df957 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:57.129501 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 277/574] 10.222.0.102 () {56 vars in 1108 bytes} [Wed Aug 24 15:33:56 2022] GET /identity/v3/roles?name=service => generated 328 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:57.130551 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05705923-7f74-4483-897f-c37644b6a9b9 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:57.138455 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a54d453-2ff3-4329-a379-e4222d4df957 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:57.140534 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a54d453-2ff3-4329-a379-e4222d4df957 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:57.160124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-7a54d453-2ff3-4329-a379-e4222d4df957 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:57.160124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:57.160124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:57.160124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:57.160124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:57.160124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:57.160124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:57.160124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:57.160124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:57.160124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:57.160124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:57.160124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:57.160124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:33:57.160124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:33:57.160124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:33:57.160124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:33:57.161284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:57.161284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:57.161284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:57.161284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:57.161284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:57.161284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:57.161284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:57.161284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:57.161284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:57.161284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:57.161284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:57.161284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:57.161284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:57.161284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:57.161284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:57.161284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:57.161284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:33:57.162379 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:33:57.162379 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:57.162379 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:57.163797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 298/575] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:33:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:57.167696 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a54d453-2ff3-4329-a379-e4222d4df957 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:57.201940 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:57.249932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05705923-7f74-4483-897f-c37644b6a9b9 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:57.288204 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a54d453-2ff3-4329-a379-e4222d4df957 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:57.291474 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3e7af9-1bf9-4364-8f0a-2a91aa059d44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:57.292211 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3e7af9-1bf9-4364-8f0a-2a91aa059d44 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:57.292599 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3e7af9-1bf9-4364-8f0a-2a91aa059d44 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:57.292991 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3e7af9-1bf9-4364-8f0a-2a91aa059d44 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:57.293750 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3e7af9-1bf9-4364-8f0a-2a91aa059d44 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:57.294215 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3e7af9-1bf9-4364-8f0a-2a91aa059d44 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:57.297437 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3e7af9-1bf9-4364-8f0a-2a91aa059d44 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:57.311742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 278/576] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:33:57 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:57.312290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3e7af9-1bf9-4364-8f0a-2a91aa059d44 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:57.330367 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928282b9-c0fd-44fc-a6dc-d122df1edb05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:57.331007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928282b9-c0fd-44fc-a6dc-d122df1edb05 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:57.331285 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928282b9-c0fd-44fc-a6dc-d122df1edb05 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:57.331562 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928282b9-c0fd-44fc-a6dc-d122df1edb05 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:57.345129 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928282b9-c0fd-44fc-a6dc-d122df1edb05 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:57.346894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928282b9-c0fd-44fc-a6dc-d122df1edb05 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:57.367577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-928282b9-c0fd-44fc-a6dc-d122df1edb05 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:57.367577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:57.367577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:57.367577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:57.367577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:57.367577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:57.367577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:57.367577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:57.367577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:57.367577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:57.367577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:57.367577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:57.367577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:33:57.367577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:33:57.367577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:33:57.367577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:33:57.368807 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:57.368807 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:57.368807 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:57.368807 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:57.368807 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:57.368807 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:57.368807 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:57.368807 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:57.368807 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:57.368807 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:57.368807 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:57.368807 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:57.368807 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:57.368807 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:57.368807 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:57.368807 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:57.368807 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:33:57.369719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:33:57.369719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:57.369719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:57.369719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 299/577] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:33:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:57.369719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928282b9-c0fd-44fc-a6dc-d122df1edb05 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:57.394864 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:57.461893 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:57.471316 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3e7af9-1bf9-4364-8f0a-2a91aa059d44 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:57.494368 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928282b9-c0fd-44fc-a6dc-d122df1edb05 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:57.517628 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3549c1-c2a2-4f20-916b-bacca1466b98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:57.518692 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3549c1-c2a2-4f20-916b-bacca1466b98 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:57.519280 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3549c1-c2a2-4f20-916b-bacca1466b98 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:57.519882 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3549c1-c2a2-4f20-916b-bacca1466b98 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:57.520702 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3549c1-c2a2-4f20-916b-bacca1466b98 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:57.521011 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3549c1-c2a2-4f20-916b-bacca1466b98 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:57.522406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3549c1-c2a2-4f20-916b-bacca1466b98 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:57.534851 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6593a900-08a8-4b17-8bc2-805b5ff837e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:57.535670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6593a900-08a8-4b17-8bc2-805b5ff837e3 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:57.535990 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6593a900-08a8-4b17-8bc2-805b5ff837e3 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:57.536267 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6593a900-08a8-4b17-8bc2-805b5ff837e3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:57.542840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 279/578] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:33:57 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:57.542840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3549c1-c2a2-4f20-916b-bacca1466b98 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:57.554957 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6593a900-08a8-4b17-8bc2-805b5ff837e3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:57.560902 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-6593a900-08a8-4b17-8bc2-805b5ff837e3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:33:57.560902 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:57.560902 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:57.560902 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:57.560902 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:57.560902 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:57.560902 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:57.560902 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:57.560902 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:57.560902 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:57.560902 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:57.560902 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:57.560902 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:57.560902 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:57.560902 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:57.560902 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:57.562044 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:57.562044 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:57.562044 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:57.562044 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:57.562044 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:57.562044 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:57.562044 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:57.562044 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:57.562044 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:57.562044 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:57.562044 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:57.562044 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:57.562044 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:57.562044 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:57.562044 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:57.562044 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:57.563350 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:57.563350 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:57.563350 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:57.563350 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:57.563350 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:57.563350 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:57.563350 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:57.563350 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:57.563350 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:57.563350 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:57.563350 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:33:57.563350 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:57.567769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 300/579] 10.222.0.102 () {56 vars in 1107 bytes} [Wed Aug 24 15:33:57 2022] GET /identity/v3/roles/service => generated 85 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:57.569184 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6593a900-08a8-4b17-8bc2-805b5ff837e3 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:57.652731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3549c1-c2a2-4f20-916b-bacca1466b98 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:57.677766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:57.678943 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6593a900-08a8-4b17-8bc2-805b5ff837e3 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:57.694229 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f17e97d-99d8-46be-b6f3-0c22fa038286 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:57.695000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f17e97d-99d8-46be-b6f3-0c22fa038286 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:57.695399 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f17e97d-99d8-46be-b6f3-0c22fa038286 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:57.695799 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f17e97d-99d8-46be-b6f3-0c22fa038286 None admin] PATH_INFO: `/v3/users/nova` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:57.704375 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f17e97d-99d8-46be-b6f3-0c22fa038286 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:57.705900 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f17e97d-99d8-46be-b6f3-0c22fa038286 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:57.715059 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cff9c9-6ad1-48c4-b4ba-698378ccd682 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:57.715611 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cff9c9-6ad1-48c4-b4ba-698378ccd682 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:57.715919 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cff9c9-6ad1-48c4-b4ba-698378ccd682 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:57.716203 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cff9c9-6ad1-48c4-b4ba-698378ccd682 None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:57.717492 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32cff9c9-6ad1-48c4-b4ba-698378ccd682 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:57.717977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32cff9c9-6ad1-48c4-b4ba-698378ccd682 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:57.719621 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32cff9c9-6ad1-48c4-b4ba-698378ccd682 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:57.721032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-6f17e97d-99d8-46be-b6f3-0c22fa038286 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 24 15:33:57.721032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:57.721032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:57.721032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:57.721032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:57.721032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:57.721032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:57.721032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:57.721032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:57.721032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:57.721032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:57.721032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:57.721032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:33:57.721032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:33:57.721032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:33:57.721032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:33:57.723386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:57.723386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:57.723386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:33:57.723386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:33:57.723386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:33:57.723386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:33:57.723386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:57.723386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:57.723386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:57.723386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:57.723386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:57.723386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:57.723386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:57.723386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:57.723386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:57.723386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:57.723386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:57.723386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:57.726608 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:57.726608 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:57.726608 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:33:57.726608 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:33:57.726608 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:33:57.726608 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:33:57.726608 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:33:57.726608 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:33:57.726608 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 24 15:33:57.726608 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:57.726608 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 280/580] 10.222.0.102 () {56 vars in 1098 bytes} [Wed Aug 24 15:33:57 2022] GET /identity/v3/users/nova => generated 82 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:57.726608 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f17e97d-99d8-46be-b6f3-0c22fa038286 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:57.735489 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:57.742278 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 301/581] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:33:57 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:57.743151 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cff9c9-6ad1-48c4-b4ba-698378ccd682 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:57.863794 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f17e97d-99d8-46be-b6f3-0c22fa038286 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:57.866077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cff9c9-6ad1-48c4-b4ba-698378ccd682 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:57.906101 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28374100-97b2-46df-ac75-de18df8489b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:57.906684 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f7e118-5e7b-40e4-8853-68b542fd0fd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:57.907008 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28374100-97b2-46df-ac75-de18df8489b7 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:57.907298 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f7e118-5e7b-40e4-8853-68b542fd0fd7 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:57.907574 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f7e118-5e7b-40e4-8853-68b542fd0fd7 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:57.907684 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28374100-97b2-46df-ac75-de18df8489b7 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:57.907891 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f7e118-5e7b-40e4-8853-68b542fd0fd7 None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:57.908163 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28374100-97b2-46df-ac75-de18df8489b7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:57.908563 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f7e118-5e7b-40e4-8853-68b542fd0fd7 None admin] RBAC: Adding query filter params (name=service) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:57.908989 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f7e118-5e7b-40e4-8853-68b542fd0fd7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:57.910327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f7e118-5e7b-40e4-8853-68b542fd0fd7 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:57.916167 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28374100-97b2-46df-ac75-de18df8489b7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:57.917660 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28374100-97b2-46df-ac75-de18df8489b7 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:57.921781 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 302/582] 10.222.0.102 () {56 vars in 1108 bytes} [Wed Aug 24 15:33:57 2022] GET /identity/v3/roles?name=service => generated 328 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:57.922795 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f7e118-5e7b-40e4-8853-68b542fd0fd7 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:57.936092 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-28374100-97b2-46df-ac75-de18df8489b7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:33:57.936092 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:57.936092 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:57.936092 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:57.936092 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:57.936092 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:57.936092 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:57.936092 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:57.936092 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:57.936092 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:57.936092 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:57.936092 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:57.936092 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:33:57.936092 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:33:57.936092 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:33:57.937040 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:33:57.937040 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:57.937040 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:57.937040 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:57.937040 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:57.937040 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:57.937040 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:57.937040 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:57.937040 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:57.937040 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:57.937040 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:57.937040 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:57.937040 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:57.937040 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:57.937040 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:57.937040 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:57.937040 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:57.938180 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:33:57.938180 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:33:57.938180 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:33:57.938180 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:33:57.938180 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:33:57.938180 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:33:57.938180 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:33:57.938180 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:57.938618 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 281/583] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:33:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:57.939603 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28374100-97b2-46df-ac75-de18df8489b7 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:57.987330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:57.992973 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:58.043767 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f7e118-5e7b-40e4-8853-68b542fd0fd7 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:58.057027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28374100-97b2-46df-ac75-de18df8489b7 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:58.081142 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3de3c1-2fc1-4ce0-aee2-1fe5addae1e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:58.081779 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3de3c1-2fc1-4ce0-aee2-1fe5addae1e0 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:58.082064 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3de3c1-2fc1-4ce0-aee2-1fe5addae1e0 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:58.082497 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3de3c1-2fc1-4ce0-aee2-1fe5addae1e0 None admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:58.083236 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e3de3c1-2fc1-4ce0-aee2-1fe5addae1e0 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:58.083547 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e3de3c1-2fc1-4ce0-aee2-1fe5addae1e0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:58.085045 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e3de3c1-2fc1-4ce0-aee2-1fe5addae1e0 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:58.098956 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b86594-21e0-41d6-9635-3c4e7f3545d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:58.099969 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b86594-21e0-41d6-9635-3c4e7f3545d9 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:58.100422 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b86594-21e0-41d6-9635-3c4e7f3545d9 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:58.100874 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b86594-21e0-41d6-9635-3c4e7f3545d9 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:58.103244 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 303/584] 10.222.0.102 () {56 vars in 1138 bytes} [Wed Aug 24 15:33:57 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 370 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:58.104764 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3de3c1-2fc1-4ce0-aee2-1fe5addae1e0 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:58.111355 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b86594-21e0-41d6-9635-3c4e7f3545d9 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:58.113953 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b86594-21e0-41d6-9635-3c4e7f3545d9 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:58.135143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-d7b86594-21e0-41d6-9635-3c4e7f3545d9 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 24 15:33:58.135143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:58.135143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:58.135143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:58.135143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:58.135143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:58.135143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:58.135143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:58.135143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:58.135143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:58.135143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:58.135143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:58.135143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:33:58.135143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:33:58.135143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:33:58.135143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:33:58.136369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:58.136369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:58.136369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:33:58.136369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:33:58.136369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:33:58.136369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:33:58.136369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:58.136369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:58.136369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:58.136369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:58.136369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:58.136369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:58.136369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:58.136369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:58.136369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:58.136369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:58.136369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:58.136369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:58.137294 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:58.137294 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:58.137294 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:33:58.137294 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:33:58.137294 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:33:58.137294 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:33:58.137294 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:33:58.137294 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:33:58.137294 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 24 15:33:58.137294 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:58.137294 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 282/585] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:33:57 2022] GET /identity/v3/users/cinder => generated 84 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:58.138272 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b86594-21e0-41d6-9635-3c4e7f3545d9 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:58.237063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3de3c1-2fc1-4ce0-aee2-1fe5addae1e0 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:58.245660 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:58.249293 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b86594-21e0-41d6-9635-3c4e7f3545d9 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:58.262486 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:58.287725 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72156ed6-4b36-4941-9601-90feaf8b6df7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:58.287971 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1983ee0-4164-49d2-8c52-8e5de09466aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:58.289905 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1983ee0-4164-49d2-8c52-8e5de09466aa None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:58.290056 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72156ed6-4b36-4941-9601-90feaf8b6df7 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:58.290137 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1983ee0-4164-49d2-8c52-8e5de09466aa None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:58.290208 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72156ed6-4b36-4941-9601-90feaf8b6df7 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:58.290283 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1983ee0-4164-49d2-8c52-8e5de09466aa None admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:58.290362 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72156ed6-4b36-4941-9601-90feaf8b6df7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:58.290444 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1983ee0-4164-49d2-8c52-8e5de09466aa None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:58.290444 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1983ee0-4164-49d2-8c52-8e5de09466aa None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:58.292753 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1983ee0-4164-49d2-8c52-8e5de09466aa None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:58.302682 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72156ed6-4b36-4941-9601-90feaf8b6df7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:58.305264 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72156ed6-4b36-4941-9601-90feaf8b6df7 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:58.311681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 283/586] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:33:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:58.316275 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1983ee0-4164-49d2-8c52-8e5de09466aa None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:58.326209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-72156ed6-4b36-4941-9601-90feaf8b6df7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:58.326209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:58.326209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:58.326209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:58.326209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:58.326209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:58.326209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:58.326209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:58.326209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:58.326209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:58.326209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:58.326209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:58.326209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:33:58.326209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:33:58.326209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:33:58.326209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:33:58.329231 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:58.329231 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:58.329231 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:58.329231 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:58.329231 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:58.329231 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:58.329231 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:58.329231 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:58.329231 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:58.329231 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:58.329231 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:58.329231 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:58.329231 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:58.329231 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:58.329231 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:58.329231 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:58.329231 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:33:58.330394 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:33:58.330394 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:58.330394 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:58.331969 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 304/587] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:33:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:58.334058 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72156ed6-4b36-4941-9601-90feaf8b6df7 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:58.430465 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1983ee0-4164-49d2-8c52-8e5de09466aa None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:58.456598 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72156ed6-4b36-4941-9601-90feaf8b6df7 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:58.475148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0011f6b3-312a-4f2d-94e7-1cf8deb85e70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:58.475700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0011f6b3-312a-4f2d-94e7-1cf8deb85e70 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:58.475983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0011f6b3-312a-4f2d-94e7-1cf8deb85e70 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:58.476255 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0011f6b3-312a-4f2d-94e7-1cf8deb85e70 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:58.489560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0011f6b3-312a-4f2d-94e7-1cf8deb85e70 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:58.491220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0011f6b3-312a-4f2d-94e7-1cf8deb85e70 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:58.501376 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001c4611-7ce6-4b9a-8a87-190b0bc1eea9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:58.502137 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001c4611-7ce6-4b9a-8a87-190b0bc1eea9 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:58.502433 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001c4611-7ce6-4b9a-8a87-190b0bc1eea9 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:58.502709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001c4611-7ce6-4b9a-8a87-190b0bc1eea9 None admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:58.503458 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001c4611-7ce6-4b9a-8a87-190b0bc1eea9 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:58.503799 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001c4611-7ce6-4b9a-8a87-190b0bc1eea9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:58.505248 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001c4611-7ce6-4b9a-8a87-190b0bc1eea9 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:58.509791 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-0011f6b3-312a-4f2d-94e7-1cf8deb85e70 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:58.509791 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:58.509791 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:58.509791 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:58.509791 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:58.509791 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:58.509791 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:58.509791 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:58.509791 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:58.509791 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:58.509791 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:58.509791 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:58.509791 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:33:58.509791 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:33:58.509791 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:33:58.509791 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:33:58.510764 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:58.510764 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:58.510764 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:58.510764 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:58.510764 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:58.510764 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:58.510764 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:58.510764 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:58.510764 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:58.510764 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:58.510764 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:58.510764 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:58.510764 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:58.510764 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:58.510764 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:58.510764 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:58.510764 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:33:58.511649 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:33:58.511649 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:58.511649 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:58.511649 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 284/588] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:33:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:58.512439 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0011f6b3-312a-4f2d-94e7-1cf8deb85e70 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:58.526959 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 305/589] 10.222.0.102 () {56 vars in 1142 bytes} [Wed Aug 24 15:33:58 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 374 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:58.527833 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001c4611-7ce6-4b9a-8a87-190b0bc1eea9 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:58.532797 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:58.569699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:58.643567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0011f6b3-312a-4f2d-94e7-1cf8deb85e70 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:58.659889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001c4611-7ce6-4b9a-8a87-190b0bc1eea9 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:58.682195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a033b4-fb7e-4137-a7bb-147436fde7d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:58.683299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a033b4-fb7e-4137-a7bb-147436fde7d3 None admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:58.683580 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a033b4-fb7e-4137-a7bb-147436fde7d3 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:58.683893 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a033b4-fb7e-4137-a7bb-147436fde7d3 None admin] PATH_INFO: `/v3/projects/3e50e6f6c1f44c648337c8d37a8e94a5/users/8c6f06186c1a4626997db5a40c31cbb9/roles/628bab09d1dc48fd9e01023b2f7671f6` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:58.701588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71984958-b06e-4ef8-a8b5-cf16c00df020 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:58.702137 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71984958-b06e-4ef8-a8b5-cf16c00df020 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:58.702415 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71984958-b06e-4ef8-a8b5-cf16c00df020 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:58.702737 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71984958-b06e-4ef8-a8b5-cf16c00df020 None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:58.703335 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71984958-b06e-4ef8-a8b5-cf16c00df020 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:58.703656 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71984958-b06e-4ef8-a8b5-cf16c00df020 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:58.705058 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71984958-b06e-4ef8-a8b5-cf16c00df020 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:58.709401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a033b4-fb7e-4137-a7bb-147436fde7d3 None admin] RBAC: Authorizing `identity:create_grant(project_id=3e50e6f6c1f44c648337c8d37a8e94a5, user_id=8c6f06186c1a4626997db5a40c31cbb9, role_id=628bab09d1dc48fd9e01023b2f7671f6)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:58.710914 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a033b4-fb7e-4137-a7bb-147436fde7d3 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:58.718500 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 306/590] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:33:58 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:58.720221 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54d4ad9-5f63-4c75-9a56-97a1758c4d86 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:58.720705 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54d4ad9-5f63-4c75-9a56-97a1758c4d86 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:58.721095 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54d4ad9-5f63-4c75-9a56-97a1758c4d86 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:58.722863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 307/591] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:33:58 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:33:58.723770 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54d4ad9-5f63-4c75-9a56-97a1758c4d86 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:58.730372 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 285/592] 10.222.0.102 () {58 vars in 1444 bytes} [Wed Aug 24 15:33:58 2022] PUT /identity/v3/projects/3e50e6f6c1f44c648337c8d37a8e94a5/users/8c6f06186c1a4626997db5a40c31cbb9/roles/628bab09d1dc48fd9e01023b2f7671f6 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:33:58.731503 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a033b4-fb7e-4137-a7bb-147436fde7d3 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:58.835195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:58.881684 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:58.902993 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54d4ad9-5f63-4c75-9a56-97a1758c4d86 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:58.916696 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a033b4-fb7e-4137-a7bb-147436fde7d3 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:58.943318 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec73a35-2ce3-42ee-99bd-1eb0702cf8c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:58.943968 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec73a35-2ce3-42ee-99bd-1eb0702cf8c2 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:58.944250 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec73a35-2ce3-42ee-99bd-1eb0702cf8c2 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:58.944523 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec73a35-2ce3-42ee-99bd-1eb0702cf8c2 None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:58.945515 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec73a35-2ce3-42ee-99bd-1eb0702cf8c2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:58.945897 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec73a35-2ce3-42ee-99bd-1eb0702cf8c2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:58.947514 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec73a35-2ce3-42ee-99bd-1eb0702cf8c2 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:58.962465 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 308/593] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:33:58 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:58.963539 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec73a35-2ce3-42ee-99bd-1eb0702cf8c2 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:58.985372 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5d4f55-cccd-4db7-8b7f-188aa6df2cc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:58.986378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5d4f55-cccd-4db7-8b7f-188aa6df2cc5 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:58.986825 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5d4f55-cccd-4db7-8b7f-188aa6df2cc5 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:58.987249 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5d4f55-cccd-4db7-8b7f-188aa6df2cc5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:59.002527 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5d4f55-cccd-4db7-8b7f-188aa6df2cc5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:59.004440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5d4f55-cccd-4db7-8b7f-188aa6df2cc5 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:59.024431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-8e5d4f55-cccd-4db7-8b7f-188aa6df2cc5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:59.024431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:59.024431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:59.024431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:59.024431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:59.024431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:59.024431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:59.024431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:59.024431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:59.024431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:59.024431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:59.024431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:59.024431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:33:59.024431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:33:59.024431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:33:59.024431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:33:59.025635 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:59.025635 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:59.025635 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:59.025635 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:59.025635 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:59.025635 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:59.025635 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:59.025635 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:59.025635 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:59.025635 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:59.025635 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:59.025635 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:59.025635 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:59.025635 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:59.025635 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:59.025635 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:59.025635 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:33:59.026731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:33:59.026731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:33:59.026731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:59.028129 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 286/594] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:33:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 297 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:59.033296 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d101c59d-0240-4e0c-a3d3-02ac8530fa35 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:59.034435 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d101c59d-0240-4e0c-a3d3-02ac8530fa35 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:59.034877 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d101c59d-0240-4e0c-a3d3-02ac8530fa35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:59.055855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-d101c59d-0240-4e0c-a3d3-02ac8530fa35 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:33:59.057745 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-d101c59d-0240-4e0c-a3d3-02ac8530fa35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:33:59.127890 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec73a35-2ce3-42ee-99bd-1eb0702cf8c2 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:59.161668 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 287/595] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:33:59 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:33:59.161668 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d101c59d-0240-4e0c-a3d3-02ac8530fa35 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:59.198382 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ba3d44-1159-44e7-b616-3b6c1061637a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:59.199141 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ba3d44-1159-44e7-b616-3b6c1061637a None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:59.199807 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ba3d44-1159-44e7-b616-3b6c1061637a None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:59.199807 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ba3d44-1159-44e7-b616-3b6c1061637a None admin] PATH_INFO: `/v3/users/neutron` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:59.212052 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64ba3d44-1159-44e7-b616-3b6c1061637a None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:59.213639 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64ba3d44-1159-44e7-b616-3b6c1061637a None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:59.228136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-64ba3d44-1159-44e7-b616-3b6c1061637a None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 24 15:33:59.228136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:59.228136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:59.228136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:59.228136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:59.228136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:59.228136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:59.228136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:59.228136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:59.228136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:59.228136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:59.228136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:59.228136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:33:59.228136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:33:59.228136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:33:59.228136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:33:59.229357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:59.229357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:59.229357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:33:59.229357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:33:59.229357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:33:59.229357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:33:59.229357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:59.229357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:59.229357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:59.229357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:59.229357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:59.229357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:59.229357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:59.229357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:59.229357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:59.229357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:59.229357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:59.229357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:59.230598 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:59.230598 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:59.230598 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:33:59.230598 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:33:59.230598 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:33:59.230598 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:33:59.230598 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:33:59.230598 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:33:59.230598 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 24 15:33:59.230598 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:59.236092 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 309/596] 10.222.0.102 () {56 vars in 1107 bytes} [Wed Aug 24 15:33:58 2022] GET /identity/v3/users/neutron => generated 85 bytes in 273 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:59.237138 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ba3d44-1159-44e7-b616-3b6c1061637a None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:59.239727 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:59.253299 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:59.330925 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d101c59d-0240-4e0c-a3d3-02ac8530fa35 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:59.374497 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd42f24f-a9ad-4b2d-bf15-46bf858311a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:59.375928 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd42f24f-a9ad-4b2d-bf15-46bf858311a5 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:59.376714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd42f24f-a9ad-4b2d-bf15-46bf858311a5 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:59.377169 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd42f24f-a9ad-4b2d-bf15-46bf858311a5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:59.385752 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ba3d44-1159-44e7-b616-3b6c1061637a None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:59.386717 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd42f24f-a9ad-4b2d-bf15-46bf858311a5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:59.388448 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd42f24f-a9ad-4b2d-bf15-46bf858311a5 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:59.402051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-fd42f24f-a9ad-4b2d-bf15-46bf858311a5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:33:59.402051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:59.402051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:59.402051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:59.402051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:59.402051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:59.402051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:59.402051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:59.402051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:59.402051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:59.402051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:59.402051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:59.402051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:33:59.402051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:33:59.402051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:33:59.403051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:33:59.403051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:59.403051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:59.403051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:59.403051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:59.403051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:59.403051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:59.403051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:59.403051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:59.403051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:59.403051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:59.403051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:59.403051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:59.403051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:59.403051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:59.403051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:59.403051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:59.403960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:33:59.403960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:33:59.403960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:33:59.403960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:33:59.403960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:33:59.403960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:33:59.403960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:33:59.403960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:59.406540 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 288/597] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:33:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:59.407576 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd42f24f-a9ad-4b2d-bf15-46bf858311a5 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:59.436829 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97580e6-2739-422d-bacf-e402af4b9b6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:59.437850 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97580e6-2739-422d-bacf-e402af4b9b6a None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:59.438479 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97580e6-2739-422d-bacf-e402af4b9b6a None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:59.438479 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97580e6-2739-422d-bacf-e402af4b9b6a None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:59.439057 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97580e6-2739-422d-bacf-e402af4b9b6a None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:59.439367 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97580e6-2739-422d-bacf-e402af4b9b6a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:59.440770 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97580e6-2739-422d-bacf-e402af4b9b6a None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:59.462596 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 310/598] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:33:59 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:59.463542 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97580e6-2739-422d-bacf-e402af4b9b6a None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:59.527299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd42f24f-a9ad-4b2d-bf15-46bf858311a5 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:59.531238 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:59.542079 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:59.582378 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97580e6-2739-422d-bacf-e402af4b9b6a None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:59.589273 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04b066d-adc0-41bb-b62f-a3514250143d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:59.589935 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04b066d-adc0-41bb-b62f-a3514250143d None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:59.590799 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04b066d-adc0-41bb-b62f-a3514250143d None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:59.591101 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04b066d-adc0-41bb-b62f-a3514250143d None admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:59.592004 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04b066d-adc0-41bb-b62f-a3514250143d None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:59.592390 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04b066d-adc0-41bb-b62f-a3514250143d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:59.594676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04b066d-adc0-41bb-b62f-a3514250143d None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:59.630359 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b782ba-b642-4e9f-9561-978f24b5478b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:59.631327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b782ba-b642-4e9f-9561-978f24b5478b None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:59.631800 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b782ba-b642-4e9f-9561-978f24b5478b None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:59.632243 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b782ba-b642-4e9f-9561-978f24b5478b None admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:59.633401 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b782ba-b642-4e9f-9561-978f24b5478b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:33:59.633910 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b782ba-b642-4e9f-9561-978f24b5478b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:59.638705 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 289/599] 10.222.0.102 () {56 vars in 1144 bytes} [Wed Aug 24 15:33:59 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 376 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:59.638705 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04b066d-adc0-41bb-b62f-a3514250143d None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:59.638922 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b782ba-b642-4e9f-9561-978f24b5478b None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:59.650570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 311/600] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:33:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:33:59.651335 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b782ba-b642-4e9f-9561-978f24b5478b None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:59.774587 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04b066d-adc0-41bb-b62f-a3514250143d None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:59.788032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b782ba-b642-4e9f-9561-978f24b5478b None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:59.816070 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee29d5df-179d-49cd-9632-782c933bb201 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:59.816710 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee29d5df-179d-49cd-9632-782c933bb201 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:59.817018 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee29d5df-179d-49cd-9632-782c933bb201 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:59.817255 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee29d5df-179d-49cd-9632-782c933bb201 None admin] PATH_INFO: `/v3/roles/service` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:59.829971 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee29d5df-179d-49cd-9632-782c933bb201 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:33:59.830620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:59.836249 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-ee29d5df-179d-49cd-9632-782c933bb201 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:33:59.836249 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:59.836249 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:59.836249 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:59.836249 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:59.836249 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:59.836249 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:59.836249 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:59.836249 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:59.836249 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:59.836249 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:59.836249 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:59.836249 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:33:59.836249 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:33:59.836249 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:33:59.836249 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:33:59.837212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:33:59.837212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:33:59.837212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:33:59.837212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:33:59.837212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:59.837212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:59.837212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:59.837212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:59.837212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:59.837212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:59.837212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:59.837212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:59.837212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:59.837212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:59.837212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:59.837212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:59.838027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:59.838027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:59.838027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:59.838027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:59.838027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:33:59.838027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:33:59.838027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:33:59.838027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:33:59.838027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:33:59.838027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:33:59.838027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:33:59.838027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:33:59.838027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 290/601] 10.222.0.102 () {56 vars in 1107 bytes} [Wed Aug 24 15:33:59 2022] GET /identity/v3/roles/service => generated 85 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:59.838721 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee29d5df-179d-49cd-9632-782c933bb201 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:59.850111 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5808d587-bdbd-412d-9d04-4ca360e4fb41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:33:59.850772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5808d587-bdbd-412d-9d04-4ca360e4fb41 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:33:59.851067 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5808d587-bdbd-412d-9d04-4ca360e4fb41 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:33:59.851345 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5808d587-bdbd-412d-9d04-4ca360e4fb41 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:33:59.859354 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5808d587-bdbd-412d-9d04-4ca360e4fb41 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:33:59.860905 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5808d587-bdbd-412d-9d04-4ca360e4fb41 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:33:59.868722 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:33:59.873241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-5808d587-bdbd-412d-9d04-4ca360e4fb41 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:33:59.873241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:33:59.873241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:33:59.873241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:33:59.873241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:33:59.873241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:33:59.873241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:33:59.873241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:33:59.873241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:33:59.873241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:33:59.873241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:33:59.873241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:33:59.873241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:33:59.873241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:33:59.873241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:33:59.874278 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:33:59.874278 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:33:59.874278 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:33:59.874278 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:33:59.874278 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:33:59.874278 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:33:59.874278 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:33:59.874278 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:33:59.874278 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:33:59.874278 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:33:59.874278 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:33:59.874278 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:33:59.874278 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:33:59.874278 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:33:59.874278 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:33:59.874278 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:33:59.874278 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:33:59.875203 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:33:59.875203 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:33:59.875203 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:33:59.875203 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:33:59.875203 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:33:59.875203 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:33:59.875203 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:33:59.875203 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:33:59.875203 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 312/602] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:33:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:33:59.875203 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5808d587-bdbd-412d-9d04-4ca360e4fb41 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:33:59.970585 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee29d5df-179d-49cd-9632-782c933bb201 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:33:59.998523 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5808d587-bdbd-412d-9d04-4ca360e4fb41 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:00.021333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a8f34f0-5e50-41e0-ba6d-0cbe1841f1b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:00.022184 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8f34f0-5e50-41e0-ba6d-0cbe1841f1b4 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:00.022558 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8f34f0-5e50-41e0-ba6d-0cbe1841f1b4 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:00.023100 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8f34f0-5e50-41e0-ba6d-0cbe1841f1b4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:00.036107 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a8f34f0-5e50-41e0-ba6d-0cbe1841f1b4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:00.038184 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a8f34f0-5e50-41e0-ba6d-0cbe1841f1b4 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:00.045422 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f10260f-b1b8-49e3-a0ac-21151699f52a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:00.046250 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f10260f-b1b8-49e3-a0ac-21151699f52a None admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:00.046528 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f10260f-b1b8-49e3-a0ac-21151699f52a None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:00.046798 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f10260f-b1b8-49e3-a0ac-21151699f52a None admin] PATH_INFO: `/v3/projects/3e50e6f6c1f44c648337c8d37a8e94a5/users/e0d544379fe44438ad10289b45124374/roles/628bab09d1dc48fd9e01023b2f7671f6` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:00.062423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-0a8f34f0-5e50-41e0-ba6d-0cbe1841f1b4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:00.062423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:00.062423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:00.062423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:00.062423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:00.062423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:00.062423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:00.062423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:00.062423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:00.062423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:00.062423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:00.062423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:00.062423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:00.062423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:00.062423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:00.062423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:00.064007 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:00.064007 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:00.064007 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:00.064007 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:00.064007 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:00.064007 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:00.064007 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:00.064007 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:00.064007 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:00.064007 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:00.064007 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:00.064007 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:00.064007 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:00.064007 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:00.064007 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:00.064007 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:00.064007 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:00.065118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:00.065118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:00.065118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:00.066241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f10260f-b1b8-49e3-a0ac-21151699f52a None admin] RBAC: Authorizing `identity:create_grant(project_id=3e50e6f6c1f44c648337c8d37a8e94a5, user_id=e0d544379fe44438ad10289b45124374, role_id=628bab09d1dc48fd9e01023b2f7671f6)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:00.067823 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f10260f-b1b8-49e3-a0ac-21151699f52a None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:00.068850 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 291/603] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:33:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:00.069737 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a8f34f0-5e50-41e0-ba6d-0cbe1841f1b4 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:00.087726 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 313/604] 10.222.0.102 () {58 vars in 1444 bytes} [Wed Aug 24 15:33:59 2022] PUT /identity/v3/projects/3e50e6f6c1f44c648337c8d37a8e94a5/users/e0d544379fe44438ad10289b45124374/roles/628bab09d1dc48fd9e01023b2f7671f6 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:34:00.088685 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f10260f-b1b8-49e3-a0ac-21151699f52a None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:00.136053 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:00.200874 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:00.237452 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a8f34f0-5e50-41e0-ba6d-0cbe1841f1b4 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:00.243343 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f10260f-b1b8-49e3-a0ac-21151699f52a None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:00.300151 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a363012-7abd-479e-be02-84567bb1074e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:00.301203 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a363012-7abd-479e-be02-84567bb1074e None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:00.301716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a363012-7abd-479e-be02-84567bb1074e None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:00.302281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a363012-7abd-479e-be02-84567bb1074e None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:00.303272 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a363012-7abd-479e-be02-84567bb1074e None admin] RBAC: Adding query filter params (name=service) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:00.305736 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a363012-7abd-479e-be02-84567bb1074e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:00.306570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f81f81-70be-45c3-bf70-a4c9bd7547dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:00.307394 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a363012-7abd-479e-be02-84567bb1074e None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:00.307582 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f81f81-70be-45c3-bf70-a4c9bd7547dd None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:00.308136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f81f81-70be-45c3-bf70-a4c9bd7547dd None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:00.308618 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f81f81-70be-45c3-bf70-a4c9bd7547dd None admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:00.310777 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f81f81-70be-45c3-bf70-a4c9bd7547dd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:00.311340 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f81f81-70be-45c3-bf70-a4c9bd7547dd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:00.315219 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f81f81-70be-45c3-bf70-a4c9bd7547dd None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:00.318840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 292/605] 10.222.0.102 () {56 vars in 1108 bytes} [Wed Aug 24 15:34:00 2022] GET /identity/v3/roles?name=service => generated 328 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:00.320522 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a363012-7abd-479e-be02-84567bb1074e None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:00.341190 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 314/606] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:34:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:00.342930 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f81f81-70be-45c3-bf70-a4c9bd7547dd None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:00.476285 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:00.482091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f81f81-70be-45c3-bf70-a4c9bd7547dd None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:00.482291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a363012-7abd-479e-be02-84567bb1074e None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:00.522956 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e6ae71-f73c-44a5-94c2-4cd5dc8a99b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:00.523982 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e6ae71-f73c-44a5-94c2-4cd5dc8a99b0 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:00.524482 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e6ae71-f73c-44a5-94c2-4cd5dc8a99b0 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:00.524958 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e6ae71-f73c-44a5-94c2-4cd5dc8a99b0 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:00.525781 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e6ae71-f73c-44a5-94c2-4cd5dc8a99b0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:00.526255 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e6ae71-f73c-44a5-94c2-4cd5dc8a99b0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:00.527928 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e6ae71-f73c-44a5-94c2-4cd5dc8a99b0 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:00.532692 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:00.533563 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6464f40-d792-4695-a726-ac6676354aaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:00.534305 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6464f40-d792-4695-a726-ac6676354aaf None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:00.534659 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6464f40-d792-4695-a726-ac6676354aaf None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:00.535024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6464f40-d792-4695-a726-ac6676354aaf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:00.550333 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6464f40-d792-4695-a726-ac6676354aaf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:00.552386 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6464f40-d792-4695-a726-ac6676354aaf None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:00.557615 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 293/607] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:00 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:00.557615 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e6ae71-f73c-44a5-94c2-4cd5dc8a99b0 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:00.573007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-f6464f40-d792-4695-a726-ac6676354aaf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:00.573007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:00.573007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:00.573007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:00.573007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:00.573007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:00.573007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:00.573007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:00.573007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:00.573007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:00.573007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:00.573007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:00.573007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:00.573007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:00.573007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:00.573007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:00.574023 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:00.574023 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:00.574023 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:00.574023 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:00.574023 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:00.574023 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:00.574023 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:00.574023 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:00.574023 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:00.574023 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:00.574023 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:00.574023 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:00.574023 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:00.574023 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:00.574023 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:00.574023 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:00.574023 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:00.575490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:00.575490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:00.575490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:00.575490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 315/608] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:00.575490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6464f40-d792-4695-a726-ac6676354aaf None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:00.681805 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e6ae71-f73c-44a5-94c2-4cd5dc8a99b0 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:00.691861 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6464f40-d792-4695-a726-ac6676354aaf None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:00.723332 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ffe193-a652-4c94-84ee-7ef8d6743b75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:00.724400 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ffe193-a652-4c94-84ee-7ef8d6743b75 None admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:00.724896 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ffe193-a652-4c94-84ee-7ef8d6743b75 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:00.725535 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ffe193-a652-4c94-84ee-7ef8d6743b75 None admin] PATH_INFO: `/v3/projects/3e50e6f6c1f44c648337c8d37a8e94a5/users/08fbd28d9593476b8753f8d64ee64f5b/roles/628bab09d1dc48fd9e01023b2f7671f6` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:00.731056 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd65a1e-6b35-4f88-b3c8-c450f8ed6198 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:00.731738 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd65a1e-6b35-4f88-b3c8-c450f8ed6198 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:00.732027 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd65a1e-6b35-4f88-b3c8-c450f8ed6198 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:00.732305 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd65a1e-6b35-4f88-b3c8-c450f8ed6198 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:00.740919 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd65a1e-6b35-4f88-b3c8-c450f8ed6198 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:00.742525 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd65a1e-6b35-4f88-b3c8-c450f8ed6198 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:00.743520 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38ffe193-a652-4c94-84ee-7ef8d6743b75 None admin] RBAC: Authorizing `identity:create_grant(project_id=3e50e6f6c1f44c648337c8d37a8e94a5, user_id=08fbd28d9593476b8753f8d64ee64f5b, role_id=628bab09d1dc48fd9e01023b2f7671f6)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:00.745399 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38ffe193-a652-4c94-84ee-7ef8d6743b75 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:00.756481 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-0cd65a1e-6b35-4f88-b3c8-c450f8ed6198 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:00.756481 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:00.756481 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:00.756481 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:00.756481 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:00.756481 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:00.756481 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:00.756481 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:00.756481 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:00.756481 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:00.756481 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:00.756481 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:00.756481 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:34:00.756481 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:34:00.756481 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:34:00.757727 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:34:00.757727 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:00.757727 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:00.757727 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:00.757727 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:00.757727 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:00.757727 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:00.757727 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:00.757727 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:00.757727 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:00.757727 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:00.757727 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:00.757727 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:00.757727 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:00.757727 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:00.757727 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:00.757727 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:00.758631 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:34:00.758631 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:34:00.758631 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:34:00.758631 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:34:00.758631 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:34:00.758631 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:34:00.758631 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:00.758631 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:00.762557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 316/609] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:34:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:00.763456 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd65a1e-6b35-4f88-b3c8-c450f8ed6198 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:00.773718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 294/610] 10.222.0.102 () {58 vars in 1444 bytes} [Wed Aug 24 15:34:00 2022] PUT /identity/v3/projects/3e50e6f6c1f44c648337c8d37a8e94a5/users/08fbd28d9593476b8753f8d64ee64f5b/roles/628bab09d1dc48fd9e01023b2f7671f6 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:34:00.775049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ffe193-a652-4c94-84ee-7ef8d6743b75 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:00.777825 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:00.848288 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:00.911901 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd65a1e-6b35-4f88-b3c8-c450f8ed6198 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:00.922017 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ffe193-a652-4c94-84ee-7ef8d6743b75 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:00.951025 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c00616-4299-4ec1-9b04-70712ffacf90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:00.951934 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c00616-4299-4ec1-9b04-70712ffacf90 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:00.952288 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c00616-4299-4ec1-9b04-70712ffacf90 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:00.952695 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c00616-4299-4ec1-9b04-70712ffacf90 None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:00.953375 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c00616-4299-4ec1-9b04-70712ffacf90 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:00.953772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c00616-4299-4ec1-9b04-70712ffacf90 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:00.955208 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c00616-4299-4ec1-9b04-70712ffacf90 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:00.968982 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72d8625-f4b4-42c6-a1ce-8f07a0839d12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:00.969613 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72d8625-f4b4-42c6-a1ce-8f07a0839d12 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:00.969880 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72d8625-f4b4-42c6-a1ce-8f07a0839d12 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:00.970159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72d8625-f4b4-42c6-a1ce-8f07a0839d12 None admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:00.970890 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 317/611] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:00 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:00.971004 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72d8625-f4b4-42c6-a1ce-8f07a0839d12 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:00.972530 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72d8625-f4b4-42c6-a1ce-8f07a0839d12 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:00.972640 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2395d42-3781-417a-b4bc-2072ba3e796c None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:00.972640 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2395d42-3781-417a-b4bc-2072ba3e796c None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:00.972640 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2395d42-3781-417a-b4bc-2072ba3e796c None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:00.973433 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72d8625-f4b4-42c6-a1ce-8f07a0839d12 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:00.974016 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 318/612] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:00 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:00.981286 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fcfda92-c72b-4ce9-9547-d3ca29b34668 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:00.982180 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fcfda92-c72b-4ce9-9547-d3ca29b34668 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:00.983180 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fcfda92-c72b-4ce9-9547-d3ca29b34668 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:00.990262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 295/613] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:34:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:00.991893 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72d8625-f4b4-42c6-a1ce-8f07a0839d12 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:01.008673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-1fcfda92-c72b-4ce9-9547-d3ca29b34668 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:01.008673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-1fcfda92-c72b-4ce9-9547-d3ca29b34668 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:01.097831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:01.105963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 319/614] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:00 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 15:34:01.106986 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fcfda92-c72b-4ce9-9547-d3ca29b34668 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:01.131292 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72d8625-f4b4-42c6-a1ce-8f07a0839d12 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:01.141379 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:01.220343 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b860ab-0ac1-4394-9ab5-c96b524f4183 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:01.221000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b860ab-0ac1-4394-9ab5-c96b524f4183 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:01.221275 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b860ab-0ac1-4394-9ab5-c96b524f4183 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:01.221555 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b860ab-0ac1-4394-9ab5-c96b524f4183 None admin] PATH_INFO: `/v3/users/swift` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:01.234095 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b860ab-0ac1-4394-9ab5-c96b524f4183 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:01.235572 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b860ab-0ac1-4394-9ab5-c96b524f4183 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:01.242173 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fcfda92-c72b-4ce9-9547-d3ca29b34668 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:01.256919 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-22b860ab-0ac1-4394-9ab5-c96b524f4183 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 24 15:34:01.256919 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:01.256919 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:01.256919 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:01.256919 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:01.256919 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:01.256919 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:01.256919 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:01.256919 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:01.256919 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:01.256919 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:01.256919 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:01.256919 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:34:01.256919 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:34:01.256919 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:34:01.256919 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:34:01.258094 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:01.258094 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:01.258094 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:34:01.258094 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:01.258094 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:34:01.258094 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:01.258094 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:01.258094 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:01.258094 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:01.258094 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:01.258094 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:01.258094 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:01.258094 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:01.258094 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:01.258094 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:01.258094 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:01.258094 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:01.258094 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:01.259293 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:01.259293 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:01.259293 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:34:01.259293 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:34:01.259293 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:34:01.259293 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:34:01.259293 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:34:01.259293 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:34:01.259293 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 24 15:34:01.259293 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:01.259293 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 296/615] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:34:00 2022] GET /identity/v3/users/swift => generated 83 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:01.265512 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b860ab-0ac1-4394-9ab5-c96b524f4183 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:01.297657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51287278-d330-4e9b-bde3-00f2f0f13abf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:01.298734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51287278-d330-4e9b-bde3-00f2f0f13abf None admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:01.299801 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51287278-d330-4e9b-bde3-00f2f0f13abf None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:01.301539 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51287278-d330-4e9b-bde3-00f2f0f13abf None admin] PATH_INFO: `/v3/projects/3e50e6f6c1f44c648337c8d37a8e94a5/users/48939674c1fd4cf599691d510fcd8858/roles/628bab09d1dc48fd9e01023b2f7671f6` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:01.336951 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51287278-d330-4e9b-bde3-00f2f0f13abf None admin] RBAC: Authorizing `identity:create_grant(project_id=3e50e6f6c1f44c648337c8d37a8e94a5, user_id=48939674c1fd4cf599691d510fcd8858, role_id=628bab09d1dc48fd9e01023b2f7671f6)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:01.338473 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51287278-d330-4e9b-bde3-00f2f0f13abf None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:01.362075 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 320/616] 10.222.0.102 () {58 vars in 1444 bytes} [Wed Aug 24 15:34:01 2022] PUT /identity/v3/projects/3e50e6f6c1f44c648337c8d37a8e94a5/users/48939674c1fd4cf599691d510fcd8858/roles/628bab09d1dc48fd9e01023b2f7671f6 => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:34:01.419619 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51287278-d330-4e9b-bde3-00f2f0f13abf None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:01.420347 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b860ab-0ac1-4394-9ab5-c96b524f4183 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:01.463860 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c32bc8-f5d0-4212-a085-0fbafd50a890 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:01.464888 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c32bc8-f5d0-4212-a085-0fbafd50a890 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:01.465334 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c32bc8-f5d0-4212-a085-0fbafd50a890 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:01.465795 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c32bc8-f5d0-4212-a085-0fbafd50a890 None admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:01.466907 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c32bc8-f5d0-4212-a085-0fbafd50a890 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:01.468102 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c32bc8-f5d0-4212-a085-0fbafd50a890 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:01.469948 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c32bc8-f5d0-4212-a085-0fbafd50a890 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:01.492313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 297/617] 10.222.0.102 () {56 vars in 1140 bytes} [Wed Aug 24 15:34:01 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 372 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:01.496891 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:01.500290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c32bc8-f5d0-4212-a085-0fbafd50a890 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:01.536700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:01.633978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51287278-d330-4e9b-bde3-00f2f0f13abf None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:01.666847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c32bc8-f5d0-4212-a085-0fbafd50a890 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:01.683055 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ecbd15-1ca9-431e-b66f-d6319171154f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:01.684562 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ecbd15-1ca9-431e-b66f-d6319171154f None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:01.684983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ecbd15-1ca9-431e-b66f-d6319171154f None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:01.685435 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ecbd15-1ca9-431e-b66f-d6319171154f None admin] PATH_INFO: `/v3/roles/service` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:01.700045 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ecbd15-1ca9-431e-b66f-d6319171154f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:34:01.707330 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-20ecbd15-1ca9-431e-b66f-d6319171154f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:34:01.707330 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:01.707330 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:01.707330 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:01.707330 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:01.707330 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:01.707330 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:01.707330 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:01.707330 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:01.707330 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:01.707330 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:01.707330 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:01.707330 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:34:01.707330 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:34:01.707330 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:34:01.707330 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:34:01.708504 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:34:01.708504 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:34:01.708504 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:34:01.708504 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:34:01.708504 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:01.708504 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:01.708504 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:01.708504 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:01.708504 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:01.708504 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:01.708504 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:01.708504 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:01.708504 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:01.708504 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:01.708504 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:01.708504 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:01.709498 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:01.709498 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:01.709498 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:01.709498 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:01.709498 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:34:01.709498 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:34:01.709498 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:34:01.709498 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:34:01.709498 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:34:01.709498 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:34:01.709498 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:34:01.709498 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:01.711887 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 321/618] 10.222.0.102 () {56 vars in 1107 bytes} [Wed Aug 24 15:34:01 2022] GET /identity/v3/roles/service => generated 85 bytes in 293 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:01.714089 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ca10de-eb16-4e4f-a362-0490110f47ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:01.714823 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ca10de-eb16-4e4f-a362-0490110f47ba None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:01.715195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ca10de-eb16-4e4f-a362-0490110f47ba None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:01.715575 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ca10de-eb16-4e4f-a362-0490110f47ba None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:01.717758 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ecbd15-1ca9-431e-b66f-d6319171154f None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:01.744772 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ca10de-eb16-4e4f-a362-0490110f47ba None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:01.746394 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ca10de-eb16-4e4f-a362-0490110f47ba None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:01.786108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-36ca10de-eb16-4e4f-a362-0490110f47ba None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:01.786108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:01.786108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:01.786108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:01.786108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:01.786108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:01.786108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:01.786108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:01.786108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:01.786108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:01.786108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:01.786108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:01.786108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:01.786108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:01.786108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:01.786108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:01.787388 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:01.787388 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:01.787388 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:01.787388 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:01.787388 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:01.787388 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:01.787388 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:01.787388 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:01.787388 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:01.787388 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:01.787388 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:01.787388 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:01.787388 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:01.787388 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:01.787388 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:01.787388 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:01.787388 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:01.788520 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:01.788520 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:01.788520 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:01.790063 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 298/619] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:01.796840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ca10de-eb16-4e4f-a362-0490110f47ba None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:01.846191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ecbd15-1ca9-431e-b66f-d6319171154f None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:01.861056 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:01.891606 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1154ec48-605f-4737-b10f-b27c90f570d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:01.892606 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1154ec48-605f-4737-b10f-b27c90f570d9 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:01.893027 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1154ec48-605f-4737-b10f-b27c90f570d9 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:01.893436 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1154ec48-605f-4737-b10f-b27c90f570d9 None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:01.894499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1154ec48-605f-4737-b10f-b27c90f570d9 None admin] RBAC: Adding query filter params (name=service) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:01.894947 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1154ec48-605f-4737-b10f-b27c90f570d9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:01.896533 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1154ec48-605f-4737-b10f-b27c90f570d9 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:01.908382 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 322/620] 10.222.0.102 () {56 vars in 1108 bytes} [Wed Aug 24 15:34:01 2022] GET /identity/v3/roles?name=service => generated 328 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:01.914117 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1154ec48-605f-4737-b10f-b27c90f570d9 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:01.946221 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:01.974839 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ca10de-eb16-4e4f-a362-0490110f47ba None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:02.016697 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c093bc-eefe-405f-968f-37e8348469f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:02.017608 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c093bc-eefe-405f-968f-37e8348469f4 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:02.018032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c093bc-eefe-405f-968f-37e8348469f4 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:02.018452 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c093bc-eefe-405f-968f-37e8348469f4 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:02.019241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c093bc-eefe-405f-968f-37e8348469f4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:02.019723 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c093bc-eefe-405f-968f-37e8348469f4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:02.021235 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c093bc-eefe-405f-968f-37e8348469f4 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:02.037138 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 299/621] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:01 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:02.046935 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c093bc-eefe-405f-968f-37e8348469f4 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:02.150686 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1154ec48-605f-4737-b10f-b27c90f570d9 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:02.218963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8895ff4-5185-4e36-a1c7-1057132782fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:02.219605 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8895ff4-5185-4e36-a1c7-1057132782fb None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:02.219904 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8895ff4-5185-4e36-a1c7-1057132782fb None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:02.220213 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8895ff4-5185-4e36-a1c7-1057132782fb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:02.230530 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c093bc-eefe-405f-968f-37e8348469f4 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:02.244032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8895ff4-5185-4e36-a1c7-1057132782fb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:02.245496 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8895ff4-5185-4e36-a1c7-1057132782fb None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:02.270888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-c8895ff4-5185-4e36-a1c7-1057132782fb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:02.270888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:02.270888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:02.270888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:02.270888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:02.270888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:02.270888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:02.270888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:02.270888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:02.270888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:02.270888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:02.270888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:02.270888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:02.270888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:02.270888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:02.270888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:02.271929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:02.271929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:02.271929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:02.271929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:02.271929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:02.271929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:02.271929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:02.271929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:02.271929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:02.271929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:02.271929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:02.271929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:02.271929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:02.271929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:02.271929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:02.271929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:02.271929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:02.272810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:02.272810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:02.272810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:02.274415 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 323/622] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 361 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:02.279777 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8895ff4-5185-4e36-a1c7-1057132782fb None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:02.300506 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6a9eed-8842-4d3c-b938-890251fc794c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:02.301133 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6a9eed-8842-4d3c-b938-890251fc794c None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:02.301411 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6a9eed-8842-4d3c-b938-890251fc794c None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:02.301682 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6a9eed-8842-4d3c-b938-890251fc794c None admin] PATH_INFO: `/v3/projects/service` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:02.313325 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:02.317854 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:02.320974 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6a9eed-8842-4d3c-b938-890251fc794c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:02.322494 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6a9eed-8842-4d3c-b938-890251fc794c None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:02.353490 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-2b6a9eed-8842-4d3c-b938-890251fc794c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:02.353490 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:02.353490 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:02.353490 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:02.353490 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:02.353490 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:02.353490 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:02.353490 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:02.353490 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:02.353490 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:02.353490 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:02.353490 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:02.353490 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:34:02.353490 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:34:02.353490 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:34:02.354462 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:34:02.354462 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:02.354462 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:02.354462 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:02.354462 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:02.354462 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:02.354462 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:02.354462 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:02.354462 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:02.354462 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:02.354462 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:02.354462 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:02.354462 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:02.354462 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:02.354462 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:02.354462 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:02.354462 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:02.355352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:34:02.355352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:34:02.355352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:34:02.355352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:34:02.355352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:34:02.355352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:34:02.355352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:02.355352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:02.363088 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 300/623] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:34:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 318 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:34:02.368704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6a9eed-8842-4d3c-b938-890251fc794c None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:02.410868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8895ff4-5185-4e36-a1c7-1057132782fb None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:02.452037 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9848f1-c805-430b-9f6c-dc75abb482cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:02.453011 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9848f1-c805-430b-9f6c-dc75abb482cf None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:02.453491 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9848f1-c805-430b-9f6c-dc75abb482cf None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:02.453993 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9848f1-c805-430b-9f6c-dc75abb482cf None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:02.454843 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd9848f1-c805-430b-9f6c-dc75abb482cf None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:02.455942 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd9848f1-c805-430b-9f6c-dc75abb482cf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:02.457525 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd9848f1-c805-430b-9f6c-dc75abb482cf None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:02.472868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 324/624] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:02 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:02.477714 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9848f1-c805-430b-9f6c-dc75abb482cf None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:02.568895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6a9eed-8842-4d3c-b938-890251fc794c None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:02.592693 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:02.614216 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9848f1-c805-430b-9f6c-dc75abb482cf None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:02.618342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb923a9b-9a12-47b0-ab05-7dd551c45a29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:02.618954 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb923a9b-9a12-47b0-ab05-7dd551c45a29 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:02.619236 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb923a9b-9a12-47b0-ab05-7dd551c45a29 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:02.619511 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb923a9b-9a12-47b0-ab05-7dd551c45a29 None admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:02.620248 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb923a9b-9a12-47b0-ab05-7dd551c45a29 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:02.620558 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb923a9b-9a12-47b0-ab05-7dd551c45a29 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:02.621993 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb923a9b-9a12-47b0-ab05-7dd551c45a29 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:02.635801 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 301/625] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:34:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:02.637080 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7395ab27-2697-4376-96ff-bb2267239358 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:02.637367 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7395ab27-2697-4376-96ff-bb2267239358 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:02.637639 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7395ab27-2697-4376-96ff-bb2267239358 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:02.644294 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 302/626] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:02 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:02.645196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7395ab27-2697-4376-96ff-bb2267239358 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:02.650695 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b5156c-2d8f-449c-aa03-010ca63b9ae1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:02.651349 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b5156c-2d8f-449c-aa03-010ca63b9ae1 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:02.651651 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b5156c-2d8f-449c-aa03-010ca63b9ae1 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:02.652036 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b5156c-2d8f-449c-aa03-010ca63b9ae1 None admin] PATH_INFO: `/v3/users/glance` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:02.661124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b5156c-2d8f-449c-aa03-010ca63b9ae1 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:02.662526 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b5156c-2d8f-449c-aa03-010ca63b9ae1 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:02.676365 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-c7b5156c-2d8f-449c-aa03-010ca63b9ae1 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 24 15:34:02.676365 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:02.676365 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:02.676365 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:02.676365 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:02.676365 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:02.676365 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:02.676365 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:02.676365 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:02.676365 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:02.676365 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:02.676365 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:02.676365 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:34:02.676365 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:34:02.676365 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:34:02.676365 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:34:02.677403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:02.677403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:02.677403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:34:02.677403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:02.677403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:34:02.677403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:02.677403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:02.677403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:02.677403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:02.677403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:02.677403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:02.677403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:02.677403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:02.677403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:02.677403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:02.677403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:02.677403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:02.677403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:02.679321 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:02.679321 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:02.679321 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:34:02.679321 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:34:02.679321 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:34:02.679321 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:34:02.679321 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:34:02.679321 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:34:02.679321 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 24 15:34:02.679321 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:02.679321 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 325/627] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:34:02 2022] GET /identity/v3/users/glance => generated 84 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:02.683710 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5f0c28-b5f0-4262-ba05-a6fd8994d56e None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:02.684095 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5f0c28-b5f0-4262-ba05-a6fd8994d56e None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:02.684314 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5f0c28-b5f0-4262-ba05-a6fd8994d56e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:02.704785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-9a5f0c28-b5f0-4262-ba05-a6fd8994d56e None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:02.705987 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-9a5f0c28-b5f0-4262-ba05-a6fd8994d56e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:02.711485 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:02.762447 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7395ab27-2697-4376-96ff-bb2267239358 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:02.785251 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 326/628] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:02 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:34:02.786121 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5f0c28-b5f0-4262-ba05-a6fd8994d56e None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:02.818941 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696ef5d0-60ad-424f-a8ac-c168b4d967f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:02.819806 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696ef5d0-60ad-424f-a8ac-c168b4d967f5 None admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:02.820200 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696ef5d0-60ad-424f-a8ac-c168b4d967f5 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:02.820582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696ef5d0-60ad-424f-a8ac-c168b4d967f5 None admin] PATH_INFO: `/v3/projects/3e50e6f6c1f44c648337c8d37a8e94a5/users/6a070d4fd8c4466c857bd03965c11349/roles/628bab09d1dc48fd9e01023b2f7671f6` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:02.860535 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696ef5d0-60ad-424f-a8ac-c168b4d967f5 None admin] RBAC: Authorizing `identity:create_grant(project_id=3e50e6f6c1f44c648337c8d37a8e94a5, user_id=6a070d4fd8c4466c857bd03965c11349, role_id=628bab09d1dc48fd9e01023b2f7671f6)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:02.862014 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696ef5d0-60ad-424f-a8ac-c168b4d967f5 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:02.886993 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:02.892987 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 303/629] 10.222.0.102 () {58 vars in 1444 bytes} [Wed Aug 24 15:34:02 2022] PUT /identity/v3/projects/3e50e6f6c1f44c648337c8d37a8e94a5/users/6a070d4fd8c4466c857bd03965c11349/roles/628bab09d1dc48fd9e01023b2f7671f6 => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:34:02.915365 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5f0c28-b5f0-4262-ba05-a6fd8994d56e None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:02.964273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15dac61-d5ed-4bd2-9cfd-ce495d9dd13c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:02.965217 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15dac61-d5ed-4bd2-9cfd-ce495d9dd13c None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:02.965651 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15dac61-d5ed-4bd2-9cfd-ce495d9dd13c None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:02.966118 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15dac61-d5ed-4bd2-9cfd-ce495d9dd13c None admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:02.967828 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f15dac61-d5ed-4bd2-9cfd-ce495d9dd13c None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:02.968296 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f15dac61-d5ed-4bd2-9cfd-ce495d9dd13c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:02.970096 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f15dac61-d5ed-4bd2-9cfd-ce495d9dd13c None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:02.991848 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 327/630] 10.222.0.102 () {56 vars in 1142 bytes} [Wed Aug 24 15:34:02 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 374 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:02.999065 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696ef5d0-60ad-424f-a8ac-c168b4d967f5 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:03.112384 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15dac61-d5ed-4bd2-9cfd-ce495d9dd13c None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:03.193970 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696ef5d0-60ad-424f-a8ac-c168b4d967f5 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:03.203230 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:03.245473 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ed17cc-22e5-4d98-86dc-c1e8b2e82d74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:03.246086 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ed17cc-22e5-4d98-86dc-c1e8b2e82d74 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:03.246367 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ed17cc-22e5-4d98-86dc-c1e8b2e82d74 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:03.246638 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ed17cc-22e5-4d98-86dc-c1e8b2e82d74 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:03.292773 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15dac61-d5ed-4bd2-9cfd-ce495d9dd13c None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:03.326009 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ed17cc-22e5-4d98-86dc-c1e8b2e82d74 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:03.327743 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ed17cc-22e5-4d98-86dc-c1e8b2e82d74 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:03.336543 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:03.344766 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2a08e3-26d1-4715-908a-1b6c9d56cee7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:03.345568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2a08e3-26d1-4715-908a-1b6c9d56cee7 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:03.346216 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2a08e3-26d1-4715-908a-1b6c9d56cee7 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:03.346600 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2a08e3-26d1-4715-908a-1b6c9d56cee7 None admin] PATH_INFO: `/v3/roles/service` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:03.363449 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe2a08e3-26d1-4715-908a-1b6c9d56cee7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:34:03.371531 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-fe2a08e3-26d1-4715-908a-1b6c9d56cee7 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:34:03.371531 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:03.371531 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:03.371531 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:03.371531 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:03.371531 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:03.371531 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:03.371531 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:03.371531 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:03.371531 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:03.371531 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:03.371531 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:03.371531 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:34:03.371531 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:34:03.371531 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:34:03.371531 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:34:03.372709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:34:03.372709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:34:03.372709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:34:03.372709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:34:03.372709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:03.372709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:03.372709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:03.372709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:03.372709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:03.372709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:03.372709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:03.372709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:03.372709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:03.372709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:03.372709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:03.372709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:03.373677 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:03.373677 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:03.373677 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:03.373677 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:03.373677 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:34:03.373677 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:34:03.373677 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:34:03.373677 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:34:03.373677 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:34:03.373677 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:34:03.373677 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:34:03.373677 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:03.379693 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 328/631] 10.222.0.102 () {56 vars in 1107 bytes} [Wed Aug 24 15:34:03 2022] GET /identity/v3/roles/service => generated 85 bytes in 264 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:34:03.381286 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee9029a-a139-4b0a-9791-6f05edfa682f None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:03.381573 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee9029a-a139-4b0a-9791-6f05edfa682f None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:03.381842 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee9029a-a139-4b0a-9791-6f05edfa682f None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:03.383317 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 329/632] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:03 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:03.383892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee9029a-a139-4b0a-9791-6f05edfa682f None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:03.481601 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 304/633] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:34:02 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 484 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:34:03.483064 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d8bdea-3bab-4d40-a57d-f9f76de8dc77 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:03.483354 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d8bdea-3bab-4d40-a57d-f9f76de8dc77 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:03.483652 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d8bdea-3bab-4d40-a57d-f9f76de8dc77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:03.506565 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:03.515242 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-77d8bdea-3bab-4d40-a57d-f9f76de8dc77 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:03.516718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-77d8bdea-3bab-4d40-a57d-f9f76de8dc77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:03.538568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee9029a-a139-4b0a-9791-6f05edfa682f None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:03.583110 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81b4417-e911-49d4-81cb-c58598f439b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:03.584194 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81b4417-e911-49d4-81cb-c58598f439b3 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:03.584693 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81b4417-e911-49d4-81cb-c58598f439b3 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:03.585260 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81b4417-e911-49d4-81cb-c58598f439b3 None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:03.586258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81b4417-e911-49d4-81cb-c58598f439b3 None admin] RBAC: Adding query filter params (name=service) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:03.586785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81b4417-e911-49d4-81cb-c58598f439b3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:03.588581 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81b4417-e911-49d4-81cb-c58598f439b3 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:03.599101 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 330/634] 10.222.0.102 () {56 vars in 1108 bytes} [Wed Aug 24 15:34:03 2022] GET /identity/v3/roles?name=service => generated 328 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:03.600323 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81b4417-e911-49d4-81cb-c58598f439b3 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:03.609488 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 305/635] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:03 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:34:03.610364 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d8bdea-3bab-4d40-a57d-f9f76de8dc77 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:03.676922 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:03.732443 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81b4417-e911-49d4-81cb-c58598f439b3 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:03.767241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d8bdea-3bab-4d40-a57d-f9f76de8dc77 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:03.778092 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d3c89b-90f4-49c1-b869-466b53f0557a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:03.779337 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d3c89b-90f4-49c1-b869-466b53f0557a None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:03.781045 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d3c89b-90f4-49c1-b869-466b53f0557a None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:03.781592 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d3c89b-90f4-49c1-b869-466b53f0557a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:03.798654 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d3c89b-90f4-49c1-b869-466b53f0557a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:03.801015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d3c89b-90f4-49c1-b869-466b53f0557a None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:03.808593 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c051358-2068-4f85-9ad2-0091db5871e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:03.809463 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c051358-2068-4f85-9ad2-0091db5871e7 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:03.809959 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c051358-2068-4f85-9ad2-0091db5871e7 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:03.810441 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c051358-2068-4f85-9ad2-0091db5871e7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:03.815543 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:03.822063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-f4d3c89b-90f4-49c1-b869-466b53f0557a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:03.822063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:03.822063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:03.822063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:03.822063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:03.822063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:03.822063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:03.822063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:03.822063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:03.822063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:03.822063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:03.822063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:03.822063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:03.822063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:03.822063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:03.822063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:03.823008 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:03.823008 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:03.823008 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:03.823008 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:03.823008 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:03.823008 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:03.823008 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:03.823008 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:03.823008 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:03.823008 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:03.823008 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:03.823008 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:03.823008 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:03.823008 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:03.823008 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:03.823008 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:03.823008 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:03.828467 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:03.828467 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:03.828467 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:03.828467 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 331/636] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:03.828467 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a78f27-9717-4753-9ce0-97ba6a7d9545 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:03.828467 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a78f27-9717-4753-9ce0-97ba6a7d9545 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:03.828467 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a78f27-9717-4753-9ce0-97ba6a7d9545 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:03.828467 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 332/637] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:03 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:03.832042 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a78f27-9717-4753-9ce0-97ba6a7d9545 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:03.833992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c051358-2068-4f85-9ad2-0091db5871e7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:03.837887 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c051358-2068-4f85-9ad2-0091db5871e7 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:03.857333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-0c051358-2068-4f85-9ad2-0091db5871e7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:03.857333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:03.857333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:03.857333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:03.857333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:03.857333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:03.857333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:03.857333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:03.857333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:03.857333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:03.857333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:03.857333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:03.857333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:03.857333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:03.857333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:03.857333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:03.858855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:03.858855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:03.858855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:03.858855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:03.858855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:03.858855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:03.858855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:03.858855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:03.858855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:03.858855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:03.858855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:03.858855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:03.858855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:03.858855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:03.858855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:03.858855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:03.858855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:03.859737 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:03.859737 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:03.859737 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:03.867144 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 306/638] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:03.868746 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64473bed-18de-4e28-a884-dd53895ad3e6 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:03.869278 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64473bed-18de-4e28-a884-dd53895ad3e6 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:03.869801 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64473bed-18de-4e28-a884-dd53895ad3e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:03.888954 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-64473bed-18de-4e28-a884-dd53895ad3e6 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:03.890698 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-64473bed-18de-4e28-a884-dd53895ad3e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:03.968209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a78f27-9717-4753-9ce0-97ba6a7d9545 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:03.976500 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:03.983763 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 307/639] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:03 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:34:03.983763 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64473bed-18de-4e28-a884-dd53895ad3e6 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:04.018929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b2381b-bf07-4e3a-80fc-523fcfaa87dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:04.019845 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b2381b-bf07-4e3a-80fc-523fcfaa87dd None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:04.020129 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b2381b-bf07-4e3a-80fc-523fcfaa87dd None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:04.020404 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b2381b-bf07-4e3a-80fc-523fcfaa87dd None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:04.021304 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b2381b-bf07-4e3a-80fc-523fcfaa87dd None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:04.021615 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b2381b-bf07-4e3a-80fc-523fcfaa87dd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:04.023816 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b2381b-bf07-4e3a-80fc-523fcfaa87dd None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:04.042282 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 333/640] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:03 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:04.043306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b2381b-bf07-4e3a-80fc-523fcfaa87dd None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:04.122061 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64473bed-18de-4e28-a884-dd53895ad3e6 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:04.149197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:04.170867 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34317acf-57b1-4225-a4b2-2d9cea3d4bd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:04.171890 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34317acf-57b1-4225-a4b2-2d9cea3d4bd3 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:04.172328 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34317acf-57b1-4225-a4b2-2d9cea3d4bd3 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:04.172746 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34317acf-57b1-4225-a4b2-2d9cea3d4bd3 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:04.173549 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34317acf-57b1-4225-a4b2-2d9cea3d4bd3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:04.174122 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34317acf-57b1-4225-a4b2-2d9cea3d4bd3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:04.175931 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34317acf-57b1-4225-a4b2-2d9cea3d4bd3 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:04.184728 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b2381b-bf07-4e3a-80fc-523fcfaa87dd None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:04.191005 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 308/641] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:03 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:04.192120 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34317acf-57b1-4225-a4b2-2d9cea3d4bd3 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:04.226670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f81c040-201b-44fd-b4e4-40e92c004d9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:04.227320 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f81c040-201b-44fd-b4e4-40e92c004d9c None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:04.227603 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f81c040-201b-44fd-b4e4-40e92c004d9c None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:04.227909 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f81c040-201b-44fd-b4e4-40e92c004d9c None admin] PATH_INFO: `/v3/roles/service` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:04.241993 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f81c040-201b-44fd-b4e4-40e92c004d9c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:34:04.247063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-5f81c040-201b-44fd-b4e4-40e92c004d9c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:34:04.247063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:04.247063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:04.247063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:04.247063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:04.247063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:04.247063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:04.247063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:04.247063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:04.247063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:04.247063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:04.247063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:04.247063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:34:04.247063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:34:04.247063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:34:04.247063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:34:04.248230 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:34:04.248230 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:34:04.248230 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:34:04.248230 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:34:04.248230 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:04.248230 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:04.248230 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:04.248230 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:04.248230 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:04.248230 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:04.248230 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:04.248230 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:04.248230 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:04.248230 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:04.248230 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:04.248230 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:04.249309 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:04.249309 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:04.249309 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:04.249309 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:04.249309 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:34:04.249309 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:34:04.249309 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:34:04.249309 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:34:04.249309 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:34:04.249309 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:34:04.249309 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:34:04.249309 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:04.253213 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 334/642] 10.222.0.102 () {56 vars in 1107 bytes} [Wed Aug 24 15:34:04 2022] GET /identity/v3/roles/service => generated 85 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:04.254170 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f81c040-201b-44fd-b4e4-40e92c004d9c None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:04.293906 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:04.324108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34317acf-57b1-4225-a4b2-2d9cea3d4bd3 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:04.374380 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53e8f26-a119-45ba-ac5f-aad1031b75ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:04.375357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53e8f26-a119-45ba-ac5f-aad1031b75ad None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:04.375851 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53e8f26-a119-45ba-ac5f-aad1031b75ad None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:04.376332 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53e8f26-a119-45ba-ac5f-aad1031b75ad None admin] PATH_INFO: `/v3/projects/service` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:04.384031 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53e8f26-a119-45ba-ac5f-aad1031b75ad None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:04.385611 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f81c040-201b-44fd-b4e4-40e92c004d9c None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:04.386042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53e8f26-a119-45ba-ac5f-aad1031b75ad None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:04.401509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-e53e8f26-a119-45ba-ac5f-aad1031b75ad None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:04.401509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:04.401509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:04.401509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:04.401509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:04.401509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:04.401509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:04.401509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:04.401509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:04.401509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:04.401509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:04.401509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:04.401509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:34:04.401509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:34:04.401509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:34:04.402509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:34:04.402509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:04.402509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:04.402509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:04.402509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:04.402509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:04.402509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:04.402509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:04.402509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:04.402509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:04.402509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:04.402509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:04.402509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:04.402509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:04.402509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:04.402509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:04.402509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:04.403403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:34:04.403403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:34:04.403403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:34:04.403403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:34:04.403403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:34:04.403403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:34:04.403403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:04.403403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:04.403403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 309/643] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:34:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:04.408658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53e8f26-a119-45ba-ac5f-aad1031b75ad None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:04.430416 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d068a82-0e16-4a93-9923-8d4e738a961f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:04.431429 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d068a82-0e16-4a93-9923-8d4e738a961f None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:04.431880 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d068a82-0e16-4a93-9923-8d4e738a961f None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:04.432315 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d068a82-0e16-4a93-9923-8d4e738a961f None admin] PATH_INFO: `/v3/users/nova` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:04.440640 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d068a82-0e16-4a93-9923-8d4e738a961f None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:04.442208 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d068a82-0e16-4a93-9923-8d4e738a961f None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:04.449546 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:04.456514 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-1d068a82-0e16-4a93-9923-8d4e738a961f None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 24 15:34:04.456514 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:04.456514 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:04.456514 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:04.456514 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:04.456514 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:04.456514 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:04.456514 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:04.456514 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:04.456514 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:04.456514 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:04.456514 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:04.456514 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:34:04.456514 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:34:04.456514 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:34:04.456514 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:34:04.457588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:04.457588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:04.457588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:34:04.457588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:04.457588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:34:04.457588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:04.457588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:04.457588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:04.457588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:04.457588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:04.457588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:04.457588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:04.457588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:04.457588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:04.457588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:04.457588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:04.457588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:04.457588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:04.458601 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:04.458601 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:04.458601 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:34:04.458601 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:34:04.458601 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:34:04.458601 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:34:04.458601 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:34:04.458601 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:34:04.458601 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 24 15:34:04.458601 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:04.458601 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 335/644] 10.222.0.102 () {56 vars in 1098 bytes} [Wed Aug 24 15:34:04 2022] GET /identity/v3/users/nova => generated 82 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:04.459292 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d068a82-0e16-4a93-9923-8d4e738a961f None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:04.520965 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53e8f26-a119-45ba-ac5f-aad1031b75ad None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:04.552268 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:04.559567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01ed26f-461e-4de6-af87-5a939887666d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:04.559567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01ed26f-461e-4de6-af87-5a939887666d None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:04.559567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01ed26f-461e-4de6-af87-5a939887666d None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:04.559567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01ed26f-461e-4de6-af87-5a939887666d None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:04.560176 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01ed26f-461e-4de6-af87-5a939887666d None admin] RBAC: Adding query filter params (name=service) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:04.560490 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01ed26f-461e-4de6-af87-5a939887666d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:04.562436 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01ed26f-461e-4de6-af87-5a939887666d None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:04.571119 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 310/645] 10.222.0.102 () {56 vars in 1108 bytes} [Wed Aug 24 15:34:04 2022] GET /identity/v3/roles?name=service => generated 328 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:04.571451 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d068a82-0e16-4a93-9923-8d4e738a961f None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:04.571959 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01ed26f-461e-4de6-af87-5a939887666d None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:04.611421 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2b93b5-2f58-4751-b6d9-33bd3fd20e5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:04.612257 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2b93b5-2f58-4751-b6d9-33bd3fd20e5a None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:04.612518 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2b93b5-2f58-4751-b6d9-33bd3fd20e5a None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:04.612849 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2b93b5-2f58-4751-b6d9-33bd3fd20e5a None admin] PATH_INFO: `/v3/roles/service` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:04.624722 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2b93b5-2f58-4751-b6d9-33bd3fd20e5a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:34:04.630152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-2b2b93b5-2f58-4751-b6d9-33bd3fd20e5a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:34:04.630152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:04.630152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:04.630152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:04.630152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:04.630152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:04.630152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:04.630152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:04.630152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:04.630152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:04.630152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:04.630152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:04.630152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:34:04.630152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:34:04.630152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:34:04.630152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:34:04.631308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:34:04.631308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:34:04.631308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:34:04.631308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:34:04.631308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:04.631308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:04.631308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:04.631308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:04.631308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:04.631308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:04.631308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:04.631308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:04.631308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:04.631308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:04.631308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:04.631308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:04.632404 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:04.632404 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:04.632404 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:04.632404 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:04.632404 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:34:04.632404 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:34:04.632404 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:34:04.632404 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:34:04.632404 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:34:04.632404 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:34:04.632404 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:34:04.632404 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:04.637576 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 336/646] 10.222.0.102 () {56 vars in 1107 bytes} [Wed Aug 24 15:34:04 2022] GET /identity/v3/roles/service => generated 85 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:04.638285 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2b93b5-2f58-4751-b6d9-33bd3fd20e5a None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:04.700555 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:04.704958 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01ed26f-461e-4de6-af87-5a939887666d None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:04.739700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7082e56d-eb04-4af9-8dc9-d2ac0263d8bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:04.740316 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7082e56d-eb04-4af9-8dc9-d2ac0263d8bf None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:04.740640 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7082e56d-eb04-4af9-8dc9-d2ac0263d8bf None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:04.740871 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7082e56d-eb04-4af9-8dc9-d2ac0263d8bf None admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:04.741570 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7082e56d-eb04-4af9-8dc9-d2ac0263d8bf None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:04.741882 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7082e56d-eb04-4af9-8dc9-d2ac0263d8bf None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:04.743307 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7082e56d-eb04-4af9-8dc9-d2ac0263d8bf None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:04.750594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2b93b5-2f58-4751-b6d9-33bd3fd20e5a None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:04.761630 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 311/647] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:34:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:04.761630 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7082e56d-eb04-4af9-8dc9-d2ac0263d8bf None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:04.792729 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcdf1d02-97ae-4f5b-8cc3-958c531ecb44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:04.793343 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdf1d02-97ae-4f5b-8cc3-958c531ecb44 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:04.793686 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdf1d02-97ae-4f5b-8cc3-958c531ecb44 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:04.794022 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdf1d02-97ae-4f5b-8cc3-958c531ecb44 None admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:04.794819 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcdf1d02-97ae-4f5b-8cc3-958c531ecb44 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:04.795231 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcdf1d02-97ae-4f5b-8cc3-958c531ecb44 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:04.797057 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcdf1d02-97ae-4f5b-8cc3-958c531ecb44 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:04.822954 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 337/648] 10.222.0.102 () {56 vars in 1138 bytes} [Wed Aug 24 15:34:04 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 370 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:04.824504 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcdf1d02-97ae-4f5b-8cc3-958c531ecb44 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:04.830612 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:04.877143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7082e56d-eb04-4af9-8dc9-d2ac0263d8bf None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:04.923245 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-448a534a-7a59-4687-909f-7386ec6e967b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:04.923937 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448a534a-7a59-4687-909f-7386ec6e967b None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:04.924215 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448a534a-7a59-4687-909f-7386ec6e967b None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:04.924490 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448a534a-7a59-4687-909f-7386ec6e967b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:04.935488 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-448a534a-7a59-4687-909f-7386ec6e967b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:04.937744 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-448a534a-7a59-4687-909f-7386ec6e967b None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:04.943115 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcdf1d02-97ae-4f5b-8cc3-958c531ecb44 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:04.959775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-448a534a-7a59-4687-909f-7386ec6e967b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:04.959775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:04.959775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:04.959775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:04.959775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:04.959775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:04.959775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:04.959775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:04.959775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:04.959775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:04.959775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:04.959775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:04.959775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:04.959775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:04.959775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:04.959775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:04.960829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:04.960829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:04.960829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:04.960829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:04.960829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:04.960829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:04.960829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:04.960829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:04.960829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:04.960829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:04.960829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:04.960829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:04.960829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:04.960829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:04.960829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:04.960829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:04.960829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:04.961718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:04.961718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:04.961718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:04.961718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 312/649] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:04.962463 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-448a534a-7a59-4687-909f-7386ec6e967b None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:04.978635 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:04.990282 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01c434e-5119-4eec-826a-911512bbbd0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:04.991028 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01c434e-5119-4eec-826a-911512bbbd0c None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:04.991402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01c434e-5119-4eec-826a-911512bbbd0c None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:04.991715 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01c434e-5119-4eec-826a-911512bbbd0c None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:04.992396 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01c434e-5119-4eec-826a-911512bbbd0c None admin] RBAC: Adding query filter params (name=service) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:04.992718 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01c434e-5119-4eec-826a-911512bbbd0c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:04.994218 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01c434e-5119-4eec-826a-911512bbbd0c None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:05.004120 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 338/650] 10.222.0.102 () {56 vars in 1108 bytes} [Wed Aug 24 15:34:04 2022] GET /identity/v3/roles?name=service => generated 328 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:05.005190 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01c434e-5119-4eec-826a-911512bbbd0c None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:05.079129 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:05.083089 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-448a534a-7a59-4687-909f-7386ec6e967b None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:05.119174 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01c434e-5119-4eec-826a-911512bbbd0c None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:05.121034 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9f49c8-3264-48b3-9fbd-6536678daacd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:05.121644 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9f49c8-3264-48b3-9fbd-6536678daacd None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:05.121924 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9f49c8-3264-48b3-9fbd-6536678daacd None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:05.122196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9f49c8-3264-48b3-9fbd-6536678daacd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:05.122935 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9f49c8-3264-48b3-9fbd-6536678daacd None admin] RBAC: Adding query filter params (role.id=628bab09d1dc48fd9e01023b2f7671f6, scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5, user.id=8c6f06186c1a4626997db5a40c31cbb9) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:05.123524 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9f49c8-3264-48b3-9fbd-6536678daacd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:05.124891 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9f49c8-3264-48b3-9fbd-6536678daacd None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:05.134817 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 313/651] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:34:04 2022] GET /identity/v3/role_assignments?user.id=8c6f06186c1a4626997db5a40c31cbb9&scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5&role.id=628bab09d1dc48fd9e01023b2f7671f6 => generated 610 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:05.134817 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9f49c8-3264-48b3-9fbd-6536678daacd None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:05.164936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f426d2-2927-4739-bbe5-1e7a00ae0909 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:05.165670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f426d2-2927-4739-bbe5-1e7a00ae0909 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:05.166003 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f426d2-2927-4739-bbe5-1e7a00ae0909 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:05.166288 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f426d2-2927-4739-bbe5-1e7a00ae0909 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:05.226429 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:05.243877 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f426d2-2927-4739-bbe5-1e7a00ae0909 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:05.245362 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f426d2-2927-4739-bbe5-1e7a00ae0909 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:05.253567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9f49c8-3264-48b3-9fbd-6536678daacd None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:05.298118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fce4f6d-9560-49d5-9eee-d517ceb0da3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:05.298727 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fce4f6d-9560-49d5-9eee-d517ceb0da3b None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:05.299003 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fce4f6d-9560-49d5-9eee-d517ceb0da3b None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:05.299271 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fce4f6d-9560-49d5-9eee-d517ceb0da3b None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:05.300077 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fce4f6d-9560-49d5-9eee-d517ceb0da3b None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:05.300382 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fce4f6d-9560-49d5-9eee-d517ceb0da3b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:05.301758 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fce4f6d-9560-49d5-9eee-d517ceb0da3b None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:05.316823 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 314/652] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:05 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:05.317871 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fce4f6d-9560-49d5-9eee-d517ceb0da3b None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:05.336278 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:05.381133 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 339/653] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:34:05 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:34:05.383291 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b887f2a-10b2-41c1-8295-3b7095bed0c7 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:05.383775 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b887f2a-10b2-41c1-8295-3b7095bed0c7 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:05.384189 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b887f2a-10b2-41c1-8295-3b7095bed0c7 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:05.385787 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 340/654] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:05 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:05.387004 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b887f2a-10b2-41c1-8295-3b7095bed0c7 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:05.444475 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fce4f6d-9560-49d5-9eee-d517ceb0da3b None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:05.482094 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a2ec07-476b-4e32-bfb8-f5c49de11d34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:05.483285 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a2ec07-476b-4e32-bfb8-f5c49de11d34 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:05.483285 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a2ec07-476b-4e32-bfb8-f5c49de11d34 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:05.483285 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a2ec07-476b-4e32-bfb8-f5c49de11d34 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:05.486126 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:05.495037 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a2ec07-476b-4e32-bfb8-f5c49de11d34 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:05.496586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a2ec07-476b-4e32-bfb8-f5c49de11d34 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:05.513041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-24a2ec07-476b-4e32-bfb8-f5c49de11d34 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:05.513041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:05.513041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:05.513041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:05.513041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:05.513041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:05.513041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:05.513041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:05.513041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:05.513041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:05.513041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:05.513041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:05.513041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:05.513041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:05.513041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:05.513041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:05.514859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:05.514859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:05.514859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:05.514859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:05.514859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:05.514859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:05.514859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:05.514859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:05.514859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:05.514859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:05.514859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:05.514859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:05.514859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:05.514859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:05.514859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:05.514859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:05.514859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:05.515871 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:05.515871 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:05.515871 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:05.515871 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 315/655] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:05.515871 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a2ec07-476b-4e32-bfb8-f5c49de11d34 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:05.524085 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b887f2a-10b2-41c1-8295-3b7095bed0c7 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:05.565005 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757d0410-c3f2-4ac8-bbc4-2d70eab97203 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:05.565642 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757d0410-c3f2-4ac8-bbc4-2d70eab97203 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:05.565922 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757d0410-c3f2-4ac8-bbc4-2d70eab97203 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:05.566192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757d0410-c3f2-4ac8-bbc4-2d70eab97203 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:05.573949 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757d0410-c3f2-4ac8-bbc4-2d70eab97203 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:05.575397 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757d0410-c3f2-4ac8-bbc4-2d70eab97203 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:05.588936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-757d0410-c3f2-4ac8-bbc4-2d70eab97203 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 24 15:34:05.588936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:05.588936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:05.588936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:05.588936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:05.588936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:05.588936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:05.588936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:05.588936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:05.588936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:05.588936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:05.588936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:05.588936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:34:05.588936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:34:05.588936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:34:05.588936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:34:05.589876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:05.589876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:05.589876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:34:05.589876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:05.589876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:34:05.589876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:05.589876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:05.589876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:05.589876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:05.589876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:05.589876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:05.589876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:05.589876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:05.589876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:05.589876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:05.589876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:05.589876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:05.589876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:05.590802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:05.590802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:05.590802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:34:05.590802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:34:05.590802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:34:05.590802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:34:05.590802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:34:05.590802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:34:05.590802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 24 15:34:05.590802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:05.590802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 341/656] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:34:05 2022] GET /identity/v3/users/cinder => generated 84 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:05.591391 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827743a0-e785-4592-9794-41920eeb9537 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:05.591391 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827743a0-e785-4592-9794-41920eeb9537 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:05.592152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827743a0-e785-4592-9794-41920eeb9537 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:05.610332 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-827743a0-e785-4592-9794-41920eeb9537 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:05.611525 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-827743a0-e785-4592-9794-41920eeb9537 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:05.635015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:05.636605 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a2ec07-476b-4e32-bfb8-f5c49de11d34 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:05.674409 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd194e0f-a2c7-4ac5-8d8a-d05b89049461 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:05.675125 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd194e0f-a2c7-4ac5-8d8a-d05b89049461 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:05.675463 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd194e0f-a2c7-4ac5-8d8a-d05b89049461 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:05.675790 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd194e0f-a2c7-4ac5-8d8a-d05b89049461 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:05.688561 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 342/657] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:05 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:34:05.689315 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-827743a0-e785-4592-9794-41920eeb9537 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:05.691771 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd194e0f-a2c7-4ac5-8d8a-d05b89049461 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:05.694481 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd194e0f-a2c7-4ac5-8d8a-d05b89049461 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:05.715348 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-fd194e0f-a2c7-4ac5-8d8a-d05b89049461 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:05.715348 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:05.715348 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:05.715348 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:05.715348 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:05.715348 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:05.715348 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:05.715348 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:05.715348 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:05.715348 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:05.715348 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:05.715348 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:05.715348 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:05.715348 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:05.715348 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:05.715348 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:05.716405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:05.716405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:05.716405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:05.716405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:05.716405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:05.716405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:05.716405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:05.716405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:05.716405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:05.716405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:05.716405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:05.716405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:05.716405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:05.716405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:05.716405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:05.716405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:05.716405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:05.718134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:05.718134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:05.718134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:05.720729 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 316/658] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:05.722004 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd194e0f-a2c7-4ac5-8d8a-d05b89049461 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:05.774821 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:05.806896 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-827743a0-e785-4592-9794-41920eeb9537 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:05.845394 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd194e0f-a2c7-4ac5-8d8a-d05b89049461 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:05.852837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4905fdb1-0c5a-4072-88a9-74d1a3fb7626 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:05.853571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4905fdb1-0c5a-4072-88a9-74d1a3fb7626 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:05.853953 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4905fdb1-0c5a-4072-88a9-74d1a3fb7626 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:05.854328 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4905fdb1-0c5a-4072-88a9-74d1a3fb7626 None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:05.855682 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4905fdb1-0c5a-4072-88a9-74d1a3fb7626 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:05.856392 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4905fdb1-0c5a-4072-88a9-74d1a3fb7626 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:05.859440 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4905fdb1-0c5a-4072-88a9-74d1a3fb7626 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:05.872655 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 343/659] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:05 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:05.874157 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4905fdb1-0c5a-4072-88a9-74d1a3fb7626 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:05.887562 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9d8438-fac4-4a9d-8341-7eeb476ec6e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:05.888437 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9d8438-fac4-4a9d-8341-7eeb476ec6e9 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:05.889405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9d8438-fac4-4a9d-8341-7eeb476ec6e9 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:05.890487 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9d8438-fac4-4a9d-8341-7eeb476ec6e9 None admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:05.891511 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9d8438-fac4-4a9d-8341-7eeb476ec6e9 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:05.892108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9d8438-fac4-4a9d-8341-7eeb476ec6e9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:05.893902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9d8438-fac4-4a9d-8341-7eeb476ec6e9 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:05.894971 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:05.915235 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 317/660] 10.222.0.102 () {56 vars in 1142 bytes} [Wed Aug 24 15:34:05 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 374 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:05.916489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9d8438-fac4-4a9d-8341-7eeb476ec6e9 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:05.988821 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4905fdb1-0c5a-4072-88a9-74d1a3fb7626 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:06.026561 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:06.028484 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7f63e3-5a84-4f54-b14e-c16324976228 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:06.028958 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7f63e3-5a84-4f54-b14e-c16324976228 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:06.029493 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7f63e3-5a84-4f54-b14e-c16324976228 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:06.029683 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7f63e3-5a84-4f54-b14e-c16324976228 None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:06.030611 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7f63e3-5a84-4f54-b14e-c16324976228 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:06.030937 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7f63e3-5a84-4f54-b14e-c16324976228 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:06.031862 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9d8438-fac4-4a9d-8341-7eeb476ec6e9 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:06.032509 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7f63e3-5a84-4f54-b14e-c16324976228 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:06.049745 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 344/661] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:05 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:06.049745 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7f63e3-5a84-4f54-b14e-c16324976228 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:06.079906 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd74429-77f2-4132-b9fc-601e55849229 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:06.080446 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd74429-77f2-4132-b9fc-601e55849229 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:06.080751 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd74429-77f2-4132-b9fc-601e55849229 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:06.081031 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd74429-77f2-4132-b9fc-601e55849229 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:06.088612 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd74429-77f2-4132-b9fc-601e55849229 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:06.090070 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd74429-77f2-4132-b9fc-601e55849229 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:06.104090 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-2cd74429-77f2-4132-b9fc-601e55849229 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 24 15:34:06.104090 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:06.104090 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:06.104090 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:06.104090 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:06.104090 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:06.104090 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:06.104090 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:06.104090 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:06.104090 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:06.104090 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:06.104090 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:06.104090 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:34:06.104090 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:34:06.104090 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:34:06.104090 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:34:06.105093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:06.105093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:06.105093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:34:06.105093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:06.105093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:34:06.105093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:06.105093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:06.105093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:06.105093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:06.105093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:06.105093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:06.105093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:06.105093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:06.105093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:06.105093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:06.105093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:06.105093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:06.105093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:06.106085 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:06.106085 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:06.106085 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:34:06.106085 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:34:06.106085 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:34:06.106085 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:34:06.106085 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:34:06.106085 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:34:06.106085 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 24 15:34:06.106085 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:06.106085 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 318/662] 10.222.0.102 () {56 vars in 1107 bytes} [Wed Aug 24 15:34:05 2022] GET /identity/v3/users/neutron => generated 85 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:06.106085 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd74429-77f2-4132-b9fc-601e55849229 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:06.155260 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:06.181474 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7f63e3-5a84-4f54-b14e-c16324976228 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:06.214381 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd74429-77f2-4132-b9fc-601e55849229 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:06.220481 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166873e1-ddb2-41e2-93c7-eac35cfac34b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:06.221105 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166873e1-ddb2-41e2-93c7-eac35cfac34b None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:06.221388 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166873e1-ddb2-41e2-93c7-eac35cfac34b None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:06.221672 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166873e1-ddb2-41e2-93c7-eac35cfac34b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:06.249898 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301baeb1-8e1e-468c-9670-b1a7aa3531be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:06.250537 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301baeb1-8e1e-468c-9670-b1a7aa3531be None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:06.250821 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301baeb1-8e1e-468c-9670-b1a7aa3531be None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:06.251101 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301baeb1-8e1e-468c-9670-b1a7aa3531be None admin] PATH_INFO: `/v3/roles/service` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:06.262517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301baeb1-8e1e-468c-9670-b1a7aa3531be None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:34:06.269102 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-301baeb1-8e1e-468c-9670-b1a7aa3531be None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:34:06.269102 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:06.269102 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:06.269102 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:06.269102 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:06.269102 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:06.269102 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:06.269102 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:06.269102 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:06.269102 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:06.269102 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:06.269102 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:06.269102 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:34:06.269102 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:34:06.269102 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:34:06.269102 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:34:06.270307 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:34:06.270307 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:34:06.270307 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:34:06.270307 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:34:06.270307 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:06.270307 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:06.270307 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:06.270307 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:06.270307 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:06.270307 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:06.270307 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:06.270307 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:06.270307 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:06.270307 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:06.270307 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:06.270307 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:06.271152 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:06.271152 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:06.271152 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:06.271152 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:06.271152 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:34:06.271152 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:34:06.271152 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:34:06.271152 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:34:06.271152 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:34:06.271152 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:34:06.271152 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:34:06.271152 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:06.271152 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 319/663] 10.222.0.102 () {56 vars in 1107 bytes} [Wed Aug 24 15:34:06 2022] GET /identity/v3/roles/service => generated 85 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:06.271904 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301baeb1-8e1e-468c-9670-b1a7aa3531be None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:06.297391 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:06.310191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-166873e1-ddb2-41e2-93c7-eac35cfac34b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:06.312377 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-166873e1-ddb2-41e2-93c7-eac35cfac34b None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:06.396048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301baeb1-8e1e-468c-9670-b1a7aa3531be None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:06.404663 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:06.431950 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 345/664] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:34:06 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:34:06.432641 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166873e1-ddb2-41e2-93c7-eac35cfac34b None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:06.439654 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8823c0c-0436-47aa-b5f3-08ce46b7a4de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:06.440299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8823c0c-0436-47aa-b5f3-08ce46b7a4de None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:06.440576 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8823c0c-0436-47aa-b5f3-08ce46b7a4de None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:06.440846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8823c0c-0436-47aa-b5f3-08ce46b7a4de None admin] PATH_INFO: `/v3/projects/service` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:06.448199 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8823c0c-0436-47aa-b5f3-08ce46b7a4de None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:06.449623 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8823c0c-0436-47aa-b5f3-08ce46b7a4de None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:06.461043 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-b8823c0c-0436-47aa-b5f3-08ce46b7a4de None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:06.461043 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:06.461043 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:06.461043 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:06.461043 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:06.461043 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:06.461043 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:06.461043 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:06.461043 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:06.461043 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:06.461043 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:06.461043 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:06.461043 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:34:06.461043 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:34:06.461043 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:34:06.462049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:34:06.462049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:06.462049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:06.462049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:06.462049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:06.462049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:06.462049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:06.462049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:06.462049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:06.462049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:06.462049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:06.462049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:06.462049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:06.462049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:06.462049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:06.462049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:06.462049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:06.463573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:34:06.463573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:34:06.463573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:34:06.463573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:34:06.463573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:34:06.463573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:34:06.463573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:06.463573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:06.463573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 320/665] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:34:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:06.463573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8823c0c-0436-47aa-b5f3-08ce46b7a4de None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:06.547325 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:06.550630 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166873e1-ddb2-41e2-93c7-eac35cfac34b None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:06.575077 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8823c0c-0436-47aa-b5f3-08ce46b7a4de None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:06.587794 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf703b8d-0957-47ef-9e03-45d14e57109e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:06.588401 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf703b8d-0957-47ef-9e03-45d14e57109e None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:06.588687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf703b8d-0957-47ef-9e03-45d14e57109e None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:06.588967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf703b8d-0957-47ef-9e03-45d14e57109e None admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:06.589715 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf703b8d-0957-47ef-9e03-45d14e57109e None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:06.590024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf703b8d-0957-47ef-9e03-45d14e57109e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:06.591464 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf703b8d-0957-47ef-9e03-45d14e57109e None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:06.611013 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 346/666] 10.222.0.102 () {56 vars in 1144 bytes} [Wed Aug 24 15:34:06 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 376 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:06.612195 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf703b8d-0957-47ef-9e03-45d14e57109e None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:06.619855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e00a462-902f-49fd-b88e-af7a9f3c4e31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:06.620575 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e00a462-902f-49fd-b88e-af7a9f3c4e31 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:06.620852 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e00a462-902f-49fd-b88e-af7a9f3c4e31 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:06.621126 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e00a462-902f-49fd-b88e-af7a9f3c4e31 None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:06.621797 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e00a462-902f-49fd-b88e-af7a9f3c4e31 None admin] RBAC: Adding query filter params (name=service) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:06.622101 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e00a462-902f-49fd-b88e-af7a9f3c4e31 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:06.623614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e00a462-902f-49fd-b88e-af7a9f3c4e31 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:06.633032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 321/667] 10.222.0.102 () {56 vars in 1108 bytes} [Wed Aug 24 15:34:06 2022] GET /identity/v3/roles?name=service => generated 328 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:06.633922 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e00a462-902f-49fd-b88e-af7a9f3c4e31 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:06.658381 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:06.734047 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf703b8d-0957-47ef-9e03-45d14e57109e None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:06.749338 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e00a462-902f-49fd-b88e-af7a9f3c4e31 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:06.770257 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ed4558-8057-449d-93d8-c87d48d0ea97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:06.770897 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ed4558-8057-449d-93d8-c87d48d0ea97 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:06.771181 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ed4558-8057-449d-93d8-c87d48d0ea97 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:06.771455 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ed4558-8057-449d-93d8-c87d48d0ea97 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:06.784408 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ed4558-8057-449d-93d8-c87d48d0ea97 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:06.786545 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ed4558-8057-449d-93d8-c87d48d0ea97 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:06.790450 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ced3e5f-f245-43ed-ab58-6fba7caab312 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:06.791271 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ced3e5f-f245-43ed-ab58-6fba7caab312 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:06.791781 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ced3e5f-f245-43ed-ab58-6fba7caab312 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:06.792227 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ced3e5f-f245-43ed-ab58-6fba7caab312 None admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:06.793245 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ced3e5f-f245-43ed-ab58-6fba7caab312 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:06.793725 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ced3e5f-f245-43ed-ab58-6fba7caab312 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:06.796042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ced3e5f-f245-43ed-ab58-6fba7caab312 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:06.803551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-d6ed4558-8057-449d-93d8-c87d48d0ea97 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:06.803551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:06.803551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:06.803551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:06.803551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:06.803551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:06.803551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:06.803551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:06.803551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:06.803551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:06.803551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:06.803551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:06.803551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:06.803551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:06.803551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:06.803551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:06.804568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:06.804568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:06.804568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:06.804568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:06.804568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:06.804568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:06.804568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:06.804568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:06.804568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:06.804568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:06.804568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:06.804568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:06.804568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:06.804568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:06.804568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:06.804568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:06.804568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:06.805581 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:06.805581 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:06.805581 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:06.809246 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 347/668] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:06.809866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:06.810108 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ed4558-8057-449d-93d8-c87d48d0ea97 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:06.815143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 322/669] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:34:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:06.816158 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ced3e5f-f245-43ed-ab58-6fba7caab312 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:06.924900 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:06.936644 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ced3e5f-f245-43ed-ab58-6fba7caab312 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:06.944315 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ed4558-8057-449d-93d8-c87d48d0ea97 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:06.976246 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5981375f-d4e5-4991-9b2c-98b25ac77161 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:06.976976 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5981375f-d4e5-4991-9b2c-98b25ac77161 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:06.977308 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5981375f-d4e5-4991-9b2c-98b25ac77161 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:06.977749 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5981375f-d4e5-4991-9b2c-98b25ac77161 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:06.982214 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd17db5e-893e-43f8-b0c0-8eaa42dc5ba6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:06.983555 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd17db5e-893e-43f8-b0c0-8eaa42dc5ba6 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:06.983871 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd17db5e-893e-43f8-b0c0-8eaa42dc5ba6 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:06.984148 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd17db5e-893e-43f8-b0c0-8eaa42dc5ba6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:06.992548 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5981375f-d4e5-4991-9b2c-98b25ac77161 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:06.994098 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5981375f-d4e5-4991-9b2c-98b25ac77161 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:07.009276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-5981375f-d4e5-4991-9b2c-98b25ac77161 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:07.009276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:07.009276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:07.009276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:07.009276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:07.009276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:07.009276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:07.009276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:07.009276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:07.009276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:07.009276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:07.009276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:07.009276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:07.009276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:07.009276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:07.009276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:07.010285 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:07.010285 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:07.010285 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:07.010285 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:07.010285 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:07.010285 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:07.010285 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:07.010285 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:07.010285 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:07.010285 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:07.010285 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:07.010285 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:07.010285 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:07.010285 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:07.010285 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:07.010285 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:07.010285 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:07.011156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:07.011156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:07.011156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:07.011156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 323/670] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:07.012041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5981375f-d4e5-4991-9b2c-98b25ac77161 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:07.062571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd17db5e-893e-43f8-b0c0-8eaa42dc5ba6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:07.064416 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd17db5e-893e-43f8-b0c0-8eaa42dc5ba6 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:07.078390 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:07.136519 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5981375f-d4e5-4991-9b2c-98b25ac77161 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:07.163775 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:07.179702 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3194005b-5a9c-45cd-8ede-5500e30d945f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:07.180242 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3194005b-5a9c-45cd-8ede-5500e30d945f None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:07.180524 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3194005b-5a9c-45cd-8ede-5500e30d945f None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:07.180800 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3194005b-5a9c-45cd-8ede-5500e30d945f None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:07.181451 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3194005b-5a9c-45cd-8ede-5500e30d945f None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:07.181757 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3194005b-5a9c-45cd-8ede-5500e30d945f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:07.183111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3194005b-5a9c-45cd-8ede-5500e30d945f None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:07.195952 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 348/671] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:34:06 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:34:07.195952 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd17db5e-893e-43f8-b0c0-8eaa42dc5ba6 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:07.196865 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 324/672] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:07 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:07.197693 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3194005b-5a9c-45cd-8ede-5500e30d945f None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:07.297675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd17db5e-893e-43f8-b0c0-8eaa42dc5ba6 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:07.300971 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3194005b-5a9c-45cd-8ede-5500e30d945f None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:07.317358 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:07.334647 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38664cb8-1e64-4eab-8f6e-203f3affa000 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:07.335256 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38664cb8-1e64-4eab-8f6e-203f3affa000 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:07.335535 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38664cb8-1e64-4eab-8f6e-203f3affa000 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:07.335843 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38664cb8-1e64-4eab-8f6e-203f3affa000 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:07.336579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38664cb8-1e64-4eab-8f6e-203f3affa000 None admin] RBAC: Adding query filter params (role.id=628bab09d1dc48fd9e01023b2f7671f6, scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5, user.id=e0d544379fe44438ad10289b45124374) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:07.336892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38664cb8-1e64-4eab-8f6e-203f3affa000 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:07.338338 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38664cb8-1e64-4eab-8f6e-203f3affa000 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:07.338692 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6571b14a-86e9-453b-b43c-34466d03c715 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:07.339288 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6571b14a-86e9-453b-b43c-34466d03c715 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:07.339575 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6571b14a-86e9-453b-b43c-34466d03c715 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:07.339877 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6571b14a-86e9-453b-b43c-34466d03c715 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:07.340693 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6571b14a-86e9-453b-b43c-34466d03c715 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:07.341000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6571b14a-86e9-453b-b43c-34466d03c715 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:07.342842 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6571b14a-86e9-453b-b43c-34466d03c715 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:07.344936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 349/673] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:34:07 2022] GET /identity/v3/role_assignments?user.id=e0d544379fe44438ad10289b45124374&scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5&role.id=628bab09d1dc48fd9e01023b2f7671f6 => generated 610 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:07.346196 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0afa7f-d741-422d-a9e1-89ff11cf83ed None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:07.346485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0afa7f-d741-422d-a9e1-89ff11cf83ed None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:07.346764 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0afa7f-d741-422d-a9e1-89ff11cf83ed None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:07.347996 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 350/674] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:07 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:07.348708 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0afa7f-d741-422d-a9e1-89ff11cf83ed None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:07.362434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 325/675] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:07 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:07.363320 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6571b14a-86e9-453b-b43c-34466d03c715 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:07.417959 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:07.483841 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0afa7f-d741-422d-a9e1-89ff11cf83ed None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:07.494480 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6571b14a-86e9-453b-b43c-34466d03c715 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:07.529677 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8830b8d5-d5ec-4ae6-93b7-2d1c712b28dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:07.530405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8830b8d5-d5ec-4ae6-93b7-2d1c712b28dd None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:07.530685 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8830b8d5-d5ec-4ae6-93b7-2d1c712b28dd None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:07.531011 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8830b8d5-d5ec-4ae6-93b7-2d1c712b28dd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:07.539874 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d3c415-2dd6-48eb-b708-a4a785cde36a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:07.540900 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d3c415-2dd6-48eb-b708-a4a785cde36a None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:07.541198 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d3c415-2dd6-48eb-b708-a4a785cde36a None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:07.541479 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d3c415-2dd6-48eb-b708-a4a785cde36a None admin] PATH_INFO: `/v3/projects/service` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:07.542859 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8830b8d5-d5ec-4ae6-93b7-2d1c712b28dd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:07.545684 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8830b8d5-d5ec-4ae6-93b7-2d1c712b28dd None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:07.549990 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d3c415-2dd6-48eb-b708-a4a785cde36a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:07.551589 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d3c415-2dd6-48eb-b708-a4a785cde36a None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:07.564863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-8830b8d5-d5ec-4ae6-93b7-2d1c712b28dd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:07.564863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:07.564863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:07.564863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:07.564863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:07.564863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:07.564863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:07.564863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:07.564863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:07.564863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:07.564863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:07.564863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:07.564863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:07.564863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:07.564863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:07.564863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:07.565889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:07.565889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:07.565889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:07.565889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:07.565889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:07.565889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:07.565889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:07.565889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:07.565889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:07.565889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:07.565889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:07.565889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:07.565889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:07.565889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:07.565889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:07.565889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:07.565889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:07.566791 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:07.566791 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:07.566791 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:07.566983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-f2d3c415-2dd6-48eb-b708-a4a785cde36a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:07.566983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:07.566983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:07.566983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:07.566983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:07.566983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:07.566983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:07.566983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:07.566983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:07.566983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:07.566983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:07.566983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:07.566983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:34:07.566983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:34:07.566983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:34:07.567840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:34:07.567840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:07.567840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:07.567840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:07.567840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:07.567840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:07.567840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:07.567840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:07.567840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:07.567840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:07.567840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:07.567840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:07.567840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:07.567840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:07.567840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:07.567840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:07.567840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:07.568734 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:34:07.568734 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:34:07.568734 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:34:07.568734 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:34:07.568734 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:34:07.568734 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:34:07.568734 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:07.568734 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:07.569152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 351/676] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:07.569152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4081010-efd8-4d5c-aa55-b489a4dccf7e None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:07.569152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4081010-efd8-4d5c-aa55-b489a4dccf7e None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:07.569152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4081010-efd8-4d5c-aa55-b489a4dccf7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:07.571543 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 326/677] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:34:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:07.572417 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d3c415-2dd6-48eb-b708-a4a785cde36a None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:07.588317 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-d4081010-efd8-4d5c-aa55-b489a4dccf7e None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:07.591669 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-d4081010-efd8-4d5c-aa55-b489a4dccf7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:07.610726 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:07.670858 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 352/678] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:07 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:34:07.672073 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4081010-efd8-4d5c-aa55-b489a4dccf7e None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:07.700495 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d3c415-2dd6-48eb-b708-a4a785cde36a None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:07.706258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:07.747183 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb272d17-e992-44ff-998d-ef6936fe53df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:07.748754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb272d17-e992-44ff-998d-ef6936fe53df None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:07.749900 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb272d17-e992-44ff-998d-ef6936fe53df None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:07.750388 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb272d17-e992-44ff-998d-ef6936fe53df None admin] PATH_INFO: `/v3/users/swift` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:07.758638 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb272d17-e992-44ff-998d-ef6936fe53df None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:07.760439 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb272d17-e992-44ff-998d-ef6936fe53df None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:07.774455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-fb272d17-e992-44ff-998d-ef6936fe53df None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 24 15:34:07.774455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:07.774455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:07.774455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:07.774455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:07.774455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:07.774455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:07.774455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:07.774455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:07.774455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:07.774455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:07.774455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:07.774455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:34:07.774455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:34:07.774455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:34:07.774455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:34:07.775489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:07.775489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:07.775489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:34:07.775489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:07.775489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:34:07.775489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:07.775489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:07.775489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:07.775489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:07.775489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:07.775489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:07.775489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:07.775489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:07.775489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:07.775489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:07.775489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:07.775489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:07.775489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:07.776458 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:07.776458 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:07.776458 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:34:07.776458 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:34:07.776458 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:34:07.776458 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:34:07.776458 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:34:07.776458 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:34:07.776458 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 24 15:34:07.776458 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:07.779041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 327/679] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:34:07 2022] GET /identity/v3/users/swift => generated 83 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:07.780934 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb272d17-e992-44ff-998d-ef6936fe53df None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:07.812120 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4081010-efd8-4d5c-aa55-b489a4dccf7e None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:07.855071 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a62f90-bdf7-4792-a9ab-1b88f7738070 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:07.855757 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a62f90-bdf7-4792-a9ab-1b88f7738070 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:07.856017 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a62f90-bdf7-4792-a9ab-1b88f7738070 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:07.856290 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a62f90-bdf7-4792-a9ab-1b88f7738070 None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:07.856942 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a62f90-bdf7-4792-a9ab-1b88f7738070 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:07.857302 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a62f90-bdf7-4792-a9ab-1b88f7738070 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:07.859527 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a62f90-bdf7-4792-a9ab-1b88f7738070 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:07.872229 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 353/680] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:07 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:07.873087 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a62f90-bdf7-4792-a9ab-1b88f7738070 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:07.896676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:07.911396 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb272d17-e992-44ff-998d-ef6936fe53df None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:07.951167 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ccb8bf8-e540-4820-84b1-828a42a54864 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:07.952558 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ccb8bf8-e540-4820-84b1-828a42a54864 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:07.952686 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ccb8bf8-e540-4820-84b1-828a42a54864 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:07.952997 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ccb8bf8-e540-4820-84b1-828a42a54864 None admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:07.953701 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ccb8bf8-e540-4820-84b1-828a42a54864 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:07.954013 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ccb8bf8-e540-4820-84b1-828a42a54864 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:07.955462 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ccb8bf8-e540-4820-84b1-828a42a54864 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:07.969830 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 328/681] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:34:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:07.970748 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ccb8bf8-e540-4820-84b1-828a42a54864 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:08.003692 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:08.015762 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a62f90-bdf7-4792-a9ab-1b88f7738070 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:08.055613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f7775e-debb-48cd-98fd-8df4cb61a7d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:08.056343 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f7775e-debb-48cd-98fd-8df4cb61a7d2 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:08.056715 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f7775e-debb-48cd-98fd-8df4cb61a7d2 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:08.056999 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f7775e-debb-48cd-98fd-8df4cb61a7d2 None admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:08.057742 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f7775e-debb-48cd-98fd-8df4cb61a7d2 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:08.058048 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f7775e-debb-48cd-98fd-8df4cb61a7d2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:08.059485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f7775e-debb-48cd-98fd-8df4cb61a7d2 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:08.083207 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 354/682] 10.222.0.102 () {56 vars in 1140 bytes} [Wed Aug 24 15:34:07 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 372 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:08.083207 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f7775e-debb-48cd-98fd-8df4cb61a7d2 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:08.087114 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ccb8bf8-e540-4820-84b1-828a42a54864 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:08.131920 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0f35d0-2e28-4bf7-88f4-c8f7ee3c7b7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:08.132500 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0f35d0-2e28-4bf7-88f4-c8f7ee3c7b7c None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:08.132786 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0f35d0-2e28-4bf7-88f4-c8f7ee3c7b7c None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:08.133065 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0f35d0-2e28-4bf7-88f4-c8f7ee3c7b7c None admin] PATH_INFO: `/v3/projects/service` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:08.143693 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b0f35d0-2e28-4bf7-88f4-c8f7ee3c7b7c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:08.145159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b0f35d0-2e28-4bf7-88f4-c8f7ee3c7b7c None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:08.157702 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-1b0f35d0-2e28-4bf7-88f4-c8f7ee3c7b7c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:08.157702 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:08.157702 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:08.157702 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:08.157702 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:08.157702 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:08.157702 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:08.157702 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:08.157702 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:08.157702 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:08.157702 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:08.157702 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:08.157702 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:34:08.157702 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:34:08.157702 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:34:08.158735 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:34:08.158735 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:08.158735 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:08.158735 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:08.158735 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:08.158735 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:08.158735 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:08.158735 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:08.158735 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:08.158735 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:08.158735 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:08.158735 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:08.158735 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:08.158735 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:08.158735 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:08.158735 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:08.158735 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:08.159651 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:34:08.159651 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:34:08.159651 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:34:08.159651 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:34:08.159651 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:34:08.159651 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:34:08.159651 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:08.159651 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:08.159651 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 329/683] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:34:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:08.159651 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0f35d0-2e28-4bf7-88f4-c8f7ee3c7b7c None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:08.175933 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:08.216267 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f7775e-debb-48cd-98fd-8df4cb61a7d2 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:08.255497 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ce5931-491b-42d6-b26b-c99a57dcf843 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:08.255987 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ce5931-491b-42d6-b26b-c99a57dcf843 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:08.256271 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ce5931-491b-42d6-b26b-c99a57dcf843 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:08.256548 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ce5931-491b-42d6-b26b-c99a57dcf843 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:08.266995 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0f35d0-2e28-4bf7-88f4-c8f7ee3c7b7c None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:08.268730 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ce5931-491b-42d6-b26b-c99a57dcf843 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:08.270169 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ce5931-491b-42d6-b26b-c99a57dcf843 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:08.286077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-19ce5931-491b-42d6-b26b-c99a57dcf843 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:08.286077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:08.286077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:08.286077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:08.286077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:08.286077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:08.286077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:08.286077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:08.286077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:08.286077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:08.286077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:08.286077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:08.286077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:08.286077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:08.286077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:08.286077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:08.287086 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:08.287086 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:08.287086 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:08.287086 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:08.287086 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:08.287086 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:08.287086 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:08.287086 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:08.287086 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:08.287086 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:08.287086 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:08.287086 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:08.287086 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:08.287086 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:08.287086 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:08.287086 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:08.287086 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:08.288014 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:08.288014 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:08.288014 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:08.288014 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 355/684] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:08.288014 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ce5931-491b-42d6-b26b-c99a57dcf843 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:08.294476 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:08.309415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9b56cc-3cdc-48af-94ca-79f18bfebbc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:08.310070 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9b56cc-3cdc-48af-94ca-79f18bfebbc5 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:08.310288 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9b56cc-3cdc-48af-94ca-79f18bfebbc5 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:08.310535 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9b56cc-3cdc-48af-94ca-79f18bfebbc5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:08.311812 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9b56cc-3cdc-48af-94ca-79f18bfebbc5 None admin] RBAC: Adding query filter params (role.id=628bab09d1dc48fd9e01023b2f7671f6, scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5, user.id=08fbd28d9593476b8753f8d64ee64f5b) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:08.312121 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9b56cc-3cdc-48af-94ca-79f18bfebbc5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:08.313809 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9b56cc-3cdc-48af-94ca-79f18bfebbc5 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:08.327661 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 330/685] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:34:08 2022] GET /identity/v3/role_assignments?user.id=08fbd28d9593476b8753f8d64ee64f5b&scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5&role.id=628bab09d1dc48fd9e01023b2f7671f6 => generated 610 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:08.327661 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9b56cc-3cdc-48af-94ca-79f18bfebbc5 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:08.432046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ce5931-491b-42d6-b26b-c99a57dcf843 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:08.436658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:08.471470 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28abc5b3-c123-40c5-b5b8-93d1e674cdb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:08.472116 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28abc5b3-c123-40c5-b5b8-93d1e674cdb3 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:08.472397 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28abc5b3-c123-40c5-b5b8-93d1e674cdb3 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:08.472707 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28abc5b3-c123-40c5-b5b8-93d1e674cdb3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:08.474127 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9b56cc-3cdc-48af-94ca-79f18bfebbc5 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:08.522202 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b6b629-4c8b-41f0-a993-ceef37e4708c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:08.523143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b6b629-4c8b-41f0-a993-ceef37e4708c None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:08.523932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b6b629-4c8b-41f0-a993-ceef37e4708c None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:08.525343 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b6b629-4c8b-41f0-a993-ceef37e4708c None admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:08.526297 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b6b629-4c8b-41f0-a993-ceef37e4708c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:08.526825 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b6b629-4c8b-41f0-a993-ceef37e4708c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:08.528586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b6b629-4c8b-41f0-a993-ceef37e4708c None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:08.545375 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 331/686] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:34:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:08.546634 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b6b629-4c8b-41f0-a993-ceef37e4708c None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:08.570388 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28abc5b3-c123-40c5-b5b8-93d1e674cdb3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:08.571914 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28abc5b3-c123-40c5-b5b8-93d1e674cdb3 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:08.574380 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:08.665732 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b6b629-4c8b-41f0-a993-ceef37e4708c None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:08.701237 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 356/687] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:34:08 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:34:08.702349 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28abc5b3-c123-40c5-b5b8-93d1e674cdb3 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:08.710635 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60767b5-5500-4b72-b4f3-3aa5a4aee6a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:08.711273 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60767b5-5500-4b72-b4f3-3aa5a4aee6a9 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:08.711562 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60767b5-5500-4b72-b4f3-3aa5a4aee6a9 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:08.712261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60767b5-5500-4b72-b4f3-3aa5a4aee6a9 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:08.713134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60767b5-5500-4b72-b4f3-3aa5a4aee6a9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:08.714291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60767b5-5500-4b72-b4f3-3aa5a4aee6a9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:08.716391 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60767b5-5500-4b72-b4f3-3aa5a4aee6a9 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:08.729249 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:08.734951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 332/688] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:08 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:08.740302 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60767b5-5500-4b72-b4f3-3aa5a4aee6a9 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:08.826928 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28abc5b3-c123-40c5-b5b8-93d1e674cdb3 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:08.829453 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:08.878937 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981debdb-5eae-4b1e-831e-3e2cf470ce95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:08.879726 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981debdb-5eae-4b1e-831e-3e2cf470ce95 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:08.880103 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981debdb-5eae-4b1e-831e-3e2cf470ce95 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:08.880468 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981debdb-5eae-4b1e-831e-3e2cf470ce95 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:08.880731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60767b5-5500-4b72-b4f3-3aa5a4aee6a9 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:08.881414 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-981debdb-5eae-4b1e-831e-3e2cf470ce95 None admin] RBAC: Adding query filter params (role.id=628bab09d1dc48fd9e01023b2f7671f6, scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5, user.id=48939674c1fd4cf599691d510fcd8858) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:08.881820 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-981debdb-5eae-4b1e-831e-3e2cf470ce95 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:08.883850 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-981debdb-5eae-4b1e-831e-3e2cf470ce95 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:08.891049 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 357/689] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:34:08 2022] GET /identity/v3/role_assignments?user.id=48939674c1fd4cf599691d510fcd8858&scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5&role.id=628bab09d1dc48fd9e01023b2f7671f6 => generated 610 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:08.892203 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981debdb-5eae-4b1e-831e-3e2cf470ce95 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:08.969921 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa5fedd-0005-4928-97f5-478176656390 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:08.970587 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa5fedd-0005-4928-97f5-478176656390 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:08.970862 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa5fedd-0005-4928-97f5-478176656390 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:08.971136 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa5fedd-0005-4928-97f5-478176656390 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:08.984621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa5fedd-0005-4928-97f5-478176656390 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:08.986205 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa5fedd-0005-4928-97f5-478176656390 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:09.008224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-7fa5fedd-0005-4928-97f5-478176656390 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:09.008224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:09.008224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:09.008224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:09.008224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:09.008224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:09.008224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:09.008224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:09.008224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:09.008224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:09.008224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:09.008224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:09.008224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:09.008224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:09.008224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:09.008224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:09.009262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:09.009262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:09.009262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:09.009262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:09.009262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:09.009262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:09.009262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:09.009262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:09.009262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:09.009262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:09.009262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:09.009262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:09.009262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:09.009262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:09.009262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:09.009262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:09.009262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:09.010155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:09.010155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:09.010155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:09.014628 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 333/690] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:09.017035 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa5fedd-0005-4928-97f5-478176656390 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:09.084475 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981debdb-5eae-4b1e-831e-3e2cf470ce95 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:09.092614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:09.148138 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21d94b1-cad7-4f69-af2c-e64f6b8f9080 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:09.149004 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21d94b1-cad7-4f69-af2c-e64f6b8f9080 None admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:09.149433 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21d94b1-cad7-4f69-af2c-e64f6b8f9080 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:09.149966 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21d94b1-cad7-4f69-af2c-e64f6b8f9080 None admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:09.150905 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c21d94b1-cad7-4f69-af2c-e64f6b8f9080 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:09.152621 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c21d94b1-cad7-4f69-af2c-e64f6b8f9080 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:09.160288 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa5fedd-0005-4928-97f5-478176656390 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:09.180505 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 358/691] 10.222.0.102 () {60 vars in 1137 bytes} [Wed Aug 24 15:34:08 2022] POST /identity/v3/users => generated 255 bytes in 288 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:34:09.209497 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c10adb-3ea6-4fc3-a293-0e60e2fe3215 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:09.210396 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c10adb-3ea6-4fc3-a293-0e60e2fe3215 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:09.210883 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c10adb-3ea6-4fc3-a293-0e60e2fe3215 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:09.211331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c10adb-3ea6-4fc3-a293-0e60e2fe3215 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:09.212166 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96c10adb-3ea6-4fc3-a293-0e60e2fe3215 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:09.212615 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96c10adb-3ea6-4fc3-a293-0e60e2fe3215 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:09.214280 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96c10adb-3ea6-4fc3-a293-0e60e2fe3215 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:09.229827 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 334/692] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:09 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:09.235845 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21d94b1-cad7-4f69-af2c-e64f6b8f9080 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:09.268609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:09.367192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21d94b1-cad7-4f69-af2c-e64f6b8f9080 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:09.405762 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db545e74-8acd-41ab-bf1b-d298add42e6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:09.406698 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db545e74-8acd-41ab-bf1b-d298add42e6f None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:09.407172 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db545e74-8acd-41ab-bf1b-d298add42e6f None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:09.407590 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db545e74-8acd-41ab-bf1b-d298add42e6f None admin] PATH_INFO: `/v3/projects/service` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:09.415441 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db545e74-8acd-41ab-bf1b-d298add42e6f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:09.418268 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db545e74-8acd-41ab-bf1b-d298add42e6f None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:09.430745 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-db545e74-8acd-41ab-bf1b-d298add42e6f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:09.430745 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:09.430745 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:09.430745 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:09.430745 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:09.430745 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:09.430745 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:09.430745 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:09.430745 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:09.430745 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:09.430745 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:09.430745 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:09.430745 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:34:09.430745 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:34:09.430745 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:34:09.431932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:34:09.431932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:09.431932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:09.431932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:09.431932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:09.431932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:09.431932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:09.431932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:09.431932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:09.431932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:09.431932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:09.431932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:09.431932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:09.431932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:09.431932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:09.431932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:09.431932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:09.433459 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:34:09.433459 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:34:09.433459 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:34:09.433459 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:34:09.433459 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:34:09.433459 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:34:09.433459 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:09.433459 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:09.435552 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 359/693] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:34:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:09.441659 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c10adb-3ea6-4fc3-a293-0e60e2fe3215 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:09.447406 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ecb6d0-cda2-49c7-9437-1d721e2b3839 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:09.447930 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ecb6d0-cda2-49c7-9437-1d721e2b3839 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:09.448375 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ecb6d0-cda2-49c7-9437-1d721e2b3839 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:09.450131 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 360/694] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:09 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:09.460654 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19d50e4-a41a-45c2-ab04-160c6a5a3447 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:09.461157 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19d50e4-a41a-45c2-ab04-160c6a5a3447 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:09.461576 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19d50e4-a41a-45c2-ab04-160c6a5a3447 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:09.482135 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-d19d50e4-a41a-45c2-ab04-160c6a5a3447 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:09.484389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-d19d50e4-a41a-45c2-ab04-160c6a5a3447 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:09.566002 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 361/695] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:09 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:34:09.590756 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c10adb-3ea6-4fc3-a293-0e60e2fe3215 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:09.601910 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:09.630452 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694dab6b-d2ff-4bb7-97a7-8d4d8d7b5932 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:09.631334 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694dab6b-d2ff-4bb7-97a7-8d4d8d7b5932 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:09.631802 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694dab6b-d2ff-4bb7-97a7-8d4d8d7b5932 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:09.632222 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694dab6b-d2ff-4bb7-97a7-8d4d8d7b5932 None admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:09.633103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694dab6b-d2ff-4bb7-97a7-8d4d8d7b5932 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:09.633570 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694dab6b-d2ff-4bb7-97a7-8d4d8d7b5932 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:09.635342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694dab6b-d2ff-4bb7-97a7-8d4d8d7b5932 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:09.650184 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 335/696] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:34:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:09.656586 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19d50e4-a41a-45c2-ab04-160c6a5a3447 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:09.680287 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:09.797162 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19d50e4-a41a-45c2-ab04-160c6a5a3447 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:09.852345 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484ff72e-dab8-4ea7-a230-1b5ef08cf55c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:09.853053 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484ff72e-dab8-4ea7-a230-1b5ef08cf55c None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:09.853339 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484ff72e-dab8-4ea7-a230-1b5ef08cf55c None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:09.853622 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484ff72e-dab8-4ea7-a230-1b5ef08cf55c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:09.854363 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-484ff72e-dab8-4ea7-a230-1b5ef08cf55c None admin] RBAC: Adding query filter params (role.id=628bab09d1dc48fd9e01023b2f7671f6, scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5, user.id=6a070d4fd8c4466c857bd03965c11349) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:09.854679 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-484ff72e-dab8-4ea7-a230-1b5ef08cf55c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:09.856152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-484ff72e-dab8-4ea7-a230-1b5ef08cf55c None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:09.868626 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 362/697] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:34:09 2022] GET /identity/v3/role_assignments?user.id=6a070d4fd8c4466c857bd03965c11349&scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5&role.id=628bab09d1dc48fd9e01023b2f7671f6 => generated 610 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:10.133651 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694dab6b-d2ff-4bb7-97a7-8d4d8d7b5932 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:10.261938 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694dab6b-d2ff-4bb7-97a7-8d4d8d7b5932 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:10.301639 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681aa9a8-c43d-4732-ac2f-01b4cdc3da5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:10.302270 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681aa9a8-c43d-4732-ac2f-01b4cdc3da5e None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:10.302550 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681aa9a8-c43d-4732-ac2f-01b4cdc3da5e None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:10.302825 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681aa9a8-c43d-4732-ac2f-01b4cdc3da5e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:10.313740 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681aa9a8-c43d-4732-ac2f-01b4cdc3da5e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:34:10.318869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-681aa9a8-c43d-4732-ac2f-01b4cdc3da5e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:34:10.318869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:10.318869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:10.318869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:10.318869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:10.318869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:10.318869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:10.318869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:10.318869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:10.318869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:10.318869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:10.318869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:10.318869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:34:10.318869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:34:10.318869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:34:10.318869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:34:10.319855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:34:10.319855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:34:10.319855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:34:10.319855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:34:10.319855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:10.319855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:10.319855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:10.319855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:10.319855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:10.319855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:10.319855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:10.319855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:10.319855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:10.319855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:10.319855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:10.319855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:10.320859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:10.320859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:10.320859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:10.320859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:10.320859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:34:10.320859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:34:10.320859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:34:10.320859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:34:10.320859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:34:10.320859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:34:10.320859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:34:10.320859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:10.320859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 336/698] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:34:10 2022] GET /identity/v3/roles/admin => generated 83 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:10.325895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484ff72e-dab8-4ea7-a230-1b5ef08cf55c None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:10.416281 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:10.449172 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484ff72e-dab8-4ea7-a230-1b5ef08cf55c None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:10.493684 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03cf3ae-3c32-4c1b-a508-8060b2a2eab7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:10.494310 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03cf3ae-3c32-4c1b-a508-8060b2a2eab7 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:10.494596 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03cf3ae-3c32-4c1b-a508-8060b2a2eab7 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:10.494871 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03cf3ae-3c32-4c1b-a508-8060b2a2eab7 None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:10.495606 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f03cf3ae-3c32-4c1b-a508-8060b2a2eab7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:10.495968 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f03cf3ae-3c32-4c1b-a508-8060b2a2eab7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:10.497344 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f03cf3ae-3c32-4c1b-a508-8060b2a2eab7 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:10.509311 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 363/699] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:34:10 2022] GET /identity/v3/roles?name=admin => generated 341 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:10.516419 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681aa9a8-c43d-4732-ac2f-01b4cdc3da5e None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:10.548308 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:10.560644 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64c967b-5c24-476b-9e93-048600eb627e None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:10.561651 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64c967b-5c24-476b-9e93-048600eb627e None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:10.562053 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64c967b-5c24-476b-9e93-048600eb627e None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:10.564243 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 364/700] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:10 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:34:10.570368 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34cb286-679f-4cd0-aa36-8a92e1c48274 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:10.570712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34cb286-679f-4cd0-aa36-8a92e1c48274 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:10.570993 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34cb286-679f-4cd0-aa36-8a92e1c48274 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:10.594204 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-c34cb286-679f-4cd0-aa36-8a92e1c48274 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:10.595511 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-c34cb286-679f-4cd0-aa36-8a92e1c48274 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:10.690755 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681aa9a8-c43d-4732-ac2f-01b4cdc3da5e None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:10.732426 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 365/701] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:10 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 15:34:10.759857 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efa1ddb-6a57-4494-9b4d-f9561e34d6b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:10.760511 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efa1ddb-6a57-4494-9b4d-f9561e34d6b6 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:10.760792 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efa1ddb-6a57-4494-9b4d-f9561e34d6b6 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:10.761066 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efa1ddb-6a57-4494-9b4d-f9561e34d6b6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:10.774589 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1efa1ddb-6a57-4494-9b4d-f9561e34d6b6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:10.776077 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1efa1ddb-6a57-4494-9b4d-f9561e34d6b6 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:10.794336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-1efa1ddb-6a57-4494-9b4d-f9561e34d6b6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:10.794336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:10.794336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:10.794336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:10.794336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:10.794336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:10.794336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:10.794336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:10.794336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:10.794336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:10.794336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:10.794336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:10.794336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:10.794336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:10.794336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:10.794336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:10.795377 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:10.795377 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:10.795377 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:10.795377 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:10.795377 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:10.795377 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:10.795377 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:10.795377 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:10.795377 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:10.795377 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:10.795377 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:10.795377 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:10.795377 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:10.795377 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:10.795377 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:10.795377 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:10.795377 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:10.796387 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:10.796387 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:10.796387 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:10.796387 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 337/702] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 281 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:34:10.804146 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34cb286-679f-4cd0-aa36-8a92e1c48274 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:10.884907 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:10.937295 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34cb286-679f-4cd0-aa36-8a92e1c48274 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:10.974981 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323b6b85-6065-4e0b-ac5c-4c3feae4d3af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:10.975513 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323b6b85-6065-4e0b-ac5c-4c3feae4d3af None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:10.975820 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323b6b85-6065-4e0b-ac5c-4c3feae4d3af None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:10.976089 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323b6b85-6065-4e0b-ac5c-4c3feae4d3af None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:10.976733 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323b6b85-6065-4e0b-ac5c-4c3feae4d3af None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:10.977032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323b6b85-6065-4e0b-ac5c-4c3feae4d3af None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:10.978390 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323b6b85-6065-4e0b-ac5c-4c3feae4d3af None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:10.995282 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 366/703] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:10 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:11.001064 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efa1ddb-6a57-4494-9b4d-f9561e34d6b6 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:11.077973 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323b6b85-6065-4e0b-ac5c-4c3feae4d3af None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:11.123585 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:11.147252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efa1ddb-6a57-4494-9b4d-f9561e34d6b6 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:11.211138 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799872e8-6706-4c24-a956-858d0b946b1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:11.211810 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799872e8-6706-4c24-a956-858d0b946b1f None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:11.212093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799872e8-6706-4c24-a956-858d0b946b1f None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:11.212368 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799872e8-6706-4c24-a956-858d0b946b1f None admin] PATH_INFO: `/v3/users/nova` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:11.240588 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799872e8-6706-4c24-a956-858d0b946b1f None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:11.241991 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799872e8-6706-4c24-a956-858d0b946b1f None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:11.250021 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323b6b85-6065-4e0b-ac5c-4c3feae4d3af None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:11.276424 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-799872e8-6706-4c24-a956-858d0b946b1f None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 24 15:34:11.276424 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:11.276424 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:11.276424 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:11.276424 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:11.276424 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:11.276424 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:11.276424 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:11.276424 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:11.276424 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:11.276424 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:11.276424 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:11.276424 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:34:11.276424 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:34:11.276424 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:34:11.276424 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:34:11.277420 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:11.277420 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:11.277420 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:34:11.277420 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:11.277420 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:34:11.277420 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:11.277420 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:11.277420 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:11.277420 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:11.277420 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:11.277420 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:11.277420 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:11.277420 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:11.277420 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:11.277420 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:11.277420 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:11.277420 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:11.277420 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:11.278414 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:11.278414 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:11.278414 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:34:11.278414 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:34:11.278414 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:34:11.278414 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:34:11.278414 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:34:11.278414 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:34:11.278414 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 24 15:34:11.278414 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:11.280529 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 338/704] 10.222.0.102 () {56 vars in 1098 bytes} [Wed Aug 24 15:34:11 2022] GET /identity/v3/users/nova => generated 82 bytes in 277 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:11.284406 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:11.285314 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799872e8-6706-4c24-a956-858d0b946b1f None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:11.290391 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea1f3fd-14ff-419f-b709-8c7e25e3f58c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:11.290963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea1f3fd-14ff-419f-b709-8c7e25e3f58c None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:11.291239 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea1f3fd-14ff-419f-b709-8c7e25e3f58c None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:11.291544 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea1f3fd-14ff-419f-b709-8c7e25e3f58c None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:11.292161 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea1f3fd-14ff-419f-b709-8c7e25e3f58c None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:11.295759 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea1f3fd-14ff-419f-b709-8c7e25e3f58c None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:11.316186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-eea1f3fd-14ff-419f-b709-8c7e25e3f58c None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 24 15:34:11.316186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:11.316186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 15:34:11.316186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 15:34:11.316186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 15:34:11.316186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 15:34:11.316186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 15:34:11.316186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:11.316186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 24 15:34:11.316186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:11.316186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 15:34:11.316186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:11.316186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:11.316186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:11.316186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:11.316186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:11.316186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:11.317324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:11.317324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:11.317324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:11.317324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:11.317324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:11.317324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:11.317324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 15:34:11.317324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 15:34:11.317324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 15:34:11.317324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 15:34:11.317324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:11.317324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:11.317324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:11.317324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:11.317324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:11.317324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:11.318178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:11.318178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:11.318178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:11.318178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:11.318178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:11.318178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:11.318178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:11.318178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:11.318178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:11.318178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:11.318178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 15:34:11.318178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:11.318178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 24 15:34:11.318178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:11.332682 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 367/705] 10.222.0.102 () {56 vars in 1134 bytes} [Wed Aug 24 15:34:11 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:34:11.333652 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea1f3fd-14ff-419f-b709-8c7e25e3f58c None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:11.455916 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea1f3fd-14ff-419f-b709-8c7e25e3f58c None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:11.493316 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328f0eaf-a4ee-4abd-91d7-34b8537292b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:11.493853 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328f0eaf-a4ee-4abd-91d7-34b8537292b7 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:11.494132 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328f0eaf-a4ee-4abd-91d7-34b8537292b7 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:11.494404 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328f0eaf-a4ee-4abd-91d7-34b8537292b7 None admin] PATH_INFO: `/v3/services` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:11.495300 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328f0eaf-a4ee-4abd-91d7-34b8537292b7 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:11.495603 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328f0eaf-a4ee-4abd-91d7-34b8537292b7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:11.498027 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328f0eaf-a4ee-4abd-91d7-34b8537292b7 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:11.503760 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 368/706] 10.222.0.102 () {56 vars in 1129 bytes} [Wed Aug 24 15:34:11 2022] GET /identity/v3/services?name=block-storage => generated 131 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:11.509924 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328f0eaf-a4ee-4abd-91d7-34b8537292b7 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:11.584060 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:11.594376 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799872e8-6706-4c24-a956-858d0b946b1f None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:11.630953 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f83064-9d17-49ef-840b-319b09994843 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:11.631565 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f83064-9d17-49ef-840b-319b09994843 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:11.631873 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f83064-9d17-49ef-840b-319b09994843 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:11.632144 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f83064-9d17-49ef-840b-319b09994843 None admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:11.632887 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f83064-9d17-49ef-840b-319b09994843 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:11.633194 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f83064-9d17-49ef-840b-319b09994843 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:11.634625 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f83064-9d17-49ef-840b-319b09994843 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:11.636837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328f0eaf-a4ee-4abd-91d7-34b8537292b7 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:11.646784 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:11.658451 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 339/707] 10.222.0.102 () {56 vars in 1138 bytes} [Wed Aug 24 15:34:11 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 370 bytes in 373 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:11.659984 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27d46e1-828a-45ba-a01a-3dfee1abed51 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:11.660369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27d46e1-828a-45ba-a01a-3dfee1abed51 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:11.660763 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27d46e1-828a-45ba-a01a-3dfee1abed51 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:11.662377 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 340/708] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:11 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:11.669332 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27d46e1-828a-45ba-a01a-3dfee1abed51 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:11.692497 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f9d709-942c-472c-a0cf-e8a1a7c8e429 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:11.693115 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f9d709-942c-472c-a0cf-e8a1a7c8e429 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:11.693394 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f9d709-942c-472c-a0cf-e8a1a7c8e429 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:11.693666 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f9d709-942c-472c-a0cf-e8a1a7c8e429 None admin] PATH_INFO: `/v3/services` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:11.694304 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f9d709-942c-472c-a0cf-e8a1a7c8e429 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:11.694603 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f9d709-942c-472c-a0cf-e8a1a7c8e429 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:11.696594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f9d709-942c-472c-a0cf-e8a1a7c8e429 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:11.711664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 369/709] 10.222.0.102 () {56 vars in 1129 bytes} [Wed Aug 24 15:34:11 2022] GET /identity/v3/services?type=block-storage => generated 131 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:11.711664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd3bbea-80eb-40eb-ac61-7af404419d7e None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:11.711664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd3bbea-80eb-40eb-ac61-7af404419d7e None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:11.711664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd3bbea-80eb-40eb-ac61-7af404419d7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:11.733072 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-cbd3bbea-80eb-40eb-ac61-7af404419d7e None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:11.735084 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-cbd3bbea-80eb-40eb-ac61-7af404419d7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:11.819122 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27d46e1-828a-45ba-a01a-3dfee1abed51 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:11.839671 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 370/710] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:11 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:34:11.839671 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67af9fa3-743d-4bfc-af66-d637c77d8101 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:11.839671 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67af9fa3-743d-4bfc-af66-d637c77d8101 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:11.839671 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67af9fa3-743d-4bfc-af66-d637c77d8101 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:11.840075 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 371/711] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:11 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:11.851647 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6b76ab-5b93-4615-96f9-493bfdaf2f9c None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:11.852158 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6b76ab-5b93-4615-96f9-493bfdaf2f9c None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:11.854463 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6b76ab-5b93-4615-96f9-493bfdaf2f9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:11.883704 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-ca6b76ab-5b93-4615-96f9-493bfdaf2f9c None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:11.886670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-ca6b76ab-5b93-4615-96f9-493bfdaf2f9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:11.891581 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-369a8781-db49-443f-b8b1-42569f4fcaf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:11.892275 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369a8781-db49-443f-b8b1-42569f4fcaf2 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:11.892556 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369a8781-db49-443f-b8b1-42569f4fcaf2 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:11.892827 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369a8781-db49-443f-b8b1-42569f4fcaf2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:11.953284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:11.980561 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 372/712] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:11 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:34:12.050241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:12.054057 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-369a8781-db49-443f-b8b1-42569f4fcaf2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:12.061890 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-369a8781-db49-443f-b8b1-42569f4fcaf2 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:12.161176 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6b76ab-5b93-4615-96f9-493bfdaf2f9c None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:12.239702 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 341/713] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:34:11 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 571 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 15:34:12.245357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-369a8781-db49-443f-b8b1-42569f4fcaf2 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:12.329080 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6b76ab-5b93-4615-96f9-493bfdaf2f9c None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:12.371091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b408ac7e-ed5d-4b01-aea5-5cbb324499ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:12.375210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b408ac7e-ed5d-4b01-aea5-5cbb324499ff None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:12.375210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b408ac7e-ed5d-4b01-aea5-5cbb324499ff None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:12.375210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b408ac7e-ed5d-4b01-aea5-5cbb324499ff None admin] PATH_INFO: `/v3/services/network` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:12.375210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b408ac7e-ed5d-4b01-aea5-5cbb324499ff None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:12.375210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b408ac7e-ed5d-4b01-aea5-5cbb324499ff None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:12.393182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-b408ac7e-ed5d-4b01-aea5-5cbb324499ff None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 24 15:34:12.393182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:12.393182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 15:34:12.393182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 15:34:12.393182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 15:34:12.393182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 15:34:12.393182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 15:34:12.393182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:12.393182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 24 15:34:12.393182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:12.393182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 15:34:12.393182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:12.393182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:12.393182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:12.393182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:12.393182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:12.393182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:12.395128 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:12.395128 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:12.395128 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:12.395128 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:12.395128 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:12.395128 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:12.395128 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 15:34:12.395128 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 15:34:12.395128 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 15:34:12.395128 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 15:34:12.395128 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:12.395128 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:12.395128 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:12.395128 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:12.395128 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:12.395128 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:12.397894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:12.397894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:12.397894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:12.397894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:12.397894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:12.397894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:12.397894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:12.397894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:12.397894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:12.397894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:12.397894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 15:34:12.397894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:12.397894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 24 15:34:12.397894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:12.397894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 373/714] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:34:12 2022] GET /identity/v3/services/network => generated 88 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:12.399472 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b408ac7e-ed5d-4b01-aea5-5cbb324499ff None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:12.413442 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:12.423270 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-369a8781-db49-443f-b8b1-42569f4fcaf2 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:12.476634 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:12.502979 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37debdf-9817-4c57-815e-32c16f4e9d80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:12.504038 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37debdf-9817-4c57-815e-32c16f4e9d80 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:12.504515 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37debdf-9817-4c57-815e-32c16f4e9d80 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:12.504997 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37debdf-9817-4c57-815e-32c16f4e9d80 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:12.516988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d37debdf-9817-4c57-815e-32c16f4e9d80 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:12.518760 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d37debdf-9817-4c57-815e-32c16f4e9d80 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:12.539154 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-d37debdf-9817-4c57-815e-32c16f4e9d80 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:12.539154 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:12.539154 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:12.539154 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:12.539154 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:12.539154 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:12.539154 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:12.539154 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:12.539154 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:12.539154 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:12.539154 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:12.539154 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:12.539154 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:12.539154 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:12.539154 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:12.539154 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:12.541029 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:12.541029 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:12.541029 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:12.541029 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:12.541029 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:12.541029 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:12.541029 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:12.541029 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:12.541029 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:12.541029 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:12.541029 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:12.541029 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:12.541029 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:12.541029 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:12.541029 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:12.541029 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:12.541029 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:12.542073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:12.542073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:12.542073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:12.544345 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 342/715] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:12.545599 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37debdf-9817-4c57-815e-32c16f4e9d80 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:12.558528 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b408ac7e-ed5d-4b01-aea5-5cbb324499ff None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:12.612753 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30dfe4a0-d6f0-4492-9b44-4dbc5178c922 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:12.613697 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30dfe4a0-d6f0-4492-9b44-4dbc5178c922 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:12.614113 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30dfe4a0-d6f0-4492-9b44-4dbc5178c922 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:12.614527 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30dfe4a0-d6f0-4492-9b44-4dbc5178c922 None admin] PATH_INFO: `/v3/roles/service` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:12.629868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30dfe4a0-d6f0-4492-9b44-4dbc5178c922 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:34:12.634983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-30dfe4a0-d6f0-4492-9b44-4dbc5178c922 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:34:12.634983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:12.634983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:12.634983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:12.634983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:12.634983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:12.634983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:12.634983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:12.634983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:12.634983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:12.634983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:12.634983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:12.634983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:34:12.634983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:34:12.634983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:34:12.634983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:34:12.636272 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:34:12.636272 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:34:12.636272 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:34:12.636272 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:34:12.636272 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:12.636272 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:12.636272 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:12.636272 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:12.636272 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:12.636272 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:12.636272 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:12.636272 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:12.636272 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:12.636272 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:12.636272 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:12.636272 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:12.637140 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:12.637140 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:12.637140 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:12.637140 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:12.637140 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:34:12.637140 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:34:12.637140 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:34:12.637140 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:34:12.637140 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:34:12.637140 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:34:12.637140 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:34:12.637140 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:12.637140 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 374/716] 10.222.0.102 () {56 vars in 1107 bytes} [Wed Aug 24 15:34:12 2022] GET /identity/v3/roles/service => generated 85 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:12.637140 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30dfe4a0-d6f0-4492-9b44-4dbc5178c922 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:12.671305 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37debdf-9817-4c57-815e-32c16f4e9d80 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:12.708642 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f356c848-14d7-4de1-807b-c60123bed43f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:12.709249 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f356c848-14d7-4de1-807b-c60123bed43f None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:12.709527 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f356c848-14d7-4de1-807b-c60123bed43f None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:12.709799 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f356c848-14d7-4de1-807b-c60123bed43f None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:12.710425 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f356c848-14d7-4de1-807b-c60123bed43f None admin] RBAC: Adding query filter params (name=network) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:12.710729 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f356c848-14d7-4de1-807b-c60123bed43f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:12.712168 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f356c848-14d7-4de1-807b-c60123bed43f None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:12.721952 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 343/717] 10.222.0.102 () {56 vars in 1117 bytes} [Wed Aug 24 15:34:12 2022] GET /identity/v3/services?name=network => generated 125 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:12.721952 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d510edad-66dc-40e8-ba45-6e5891609afd None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:12.721952 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d510edad-66dc-40e8-ba45-6e5891609afd None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:12.721952 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d510edad-66dc-40e8-ba45-6e5891609afd None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:12.721952 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 344/718] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:12 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:12.721952 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d510edad-66dc-40e8-ba45-6e5891609afd None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:12.739594 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:12.754680 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30dfe4a0-d6f0-4492-9b44-4dbc5178c922 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:12.757249 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:12.791583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1910cb-592a-4bf9-a0b6-bb9af44bf395 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:12.792226 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1910cb-592a-4bf9-a0b6-bb9af44bf395 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:12.793042 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1910cb-592a-4bf9-a0b6-bb9af44bf395 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:12.793042 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1910cb-592a-4bf9-a0b6-bb9af44bf395 None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:12.793605 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1910cb-592a-4bf9-a0b6-bb9af44bf395 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:12.793785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1910cb-592a-4bf9-a0b6-bb9af44bf395 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:12.795052 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1910cb-592a-4bf9-a0b6-bb9af44bf395 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:12.814786 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 375/719] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:12 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:12.814786 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1910cb-592a-4bf9-a0b6-bb9af44bf395 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:12.849558 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d510edad-66dc-40e8-ba45-6e5891609afd None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:12.895102 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93dfc860-b125-4d3d-9c43-0083dc741131 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:12.896045 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93dfc860-b125-4d3d-9c43-0083dc741131 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:12.896456 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93dfc860-b125-4d3d-9c43-0083dc741131 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:12.896875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93dfc860-b125-4d3d-9c43-0083dc741131 None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:12.897985 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93dfc860-b125-4d3d-9c43-0083dc741131 None admin] RBAC: Adding query filter params (name=service) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:12.898460 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93dfc860-b125-4d3d-9c43-0083dc741131 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:12.900653 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93dfc860-b125-4d3d-9c43-0083dc741131 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:12.911397 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 345/720] 10.222.0.102 () {56 vars in 1108 bytes} [Wed Aug 24 15:34:12 2022] GET /identity/v3/roles?name=service => generated 328 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:12.912746 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc45283e-eb60-4aca-9822-4cb20cded6f1 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:12.913041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc45283e-eb60-4aca-9822-4cb20cded6f1 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:12.913348 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc45283e-eb60-4aca-9822-4cb20cded6f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:12.935709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-fc45283e-eb60-4aca-9822-4cb20cded6f1 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:12.937492 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-fc45283e-eb60-4aca-9822-4cb20cded6f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:12.948214 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1910cb-592a-4bf9-a0b6-bb9af44bf395 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:13.003756 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98bef6d8-fc48-424d-b532-65965fad69a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:13.004423 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bef6d8-fc48-424d-b532-65965fad69a8 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:13.004760 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bef6d8-fc48-424d-b532-65965fad69a8 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:13.005049 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bef6d8-fc48-424d-b532-65965fad69a8 None admin] PATH_INFO: `/v3/services` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:13.005929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98bef6d8-fc48-424d-b532-65965fad69a8 None admin] RBAC: Adding query filter params (type=network) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:13.006398 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98bef6d8-fc48-424d-b532-65965fad69a8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:13.008604 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98bef6d8-fc48-424d-b532-65965fad69a8 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:13.015855 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 376/721] 10.222.0.102 () {56 vars in 1117 bytes} [Wed Aug 24 15:34:12 2022] GET /identity/v3/services?type=network => generated 125 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:13.016697 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98bef6d8-fc48-424d-b532-65965fad69a8 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:13.055985 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 346/722] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:12 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:34:13.057032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc45283e-eb60-4aca-9822-4cb20cded6f1 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:13.067880 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:13.083381 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:13.214278 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98bef6d8-fc48-424d-b532-65965fad69a8 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:13.225424 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc45283e-eb60-4aca-9822-4cb20cded6f1 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:13.263020 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515f41c5-9194-4a47-b5f1-818206f8c344 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:13.263675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515f41c5-9194-4a47-b5f1-818206f8c344 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:13.264549 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515f41c5-9194-4a47-b5f1-818206f8c344 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:13.264944 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515f41c5-9194-4a47-b5f1-818206f8c344 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:13.268616 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c4de73-c484-46fb-9585-b5156799289c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:13.269643 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c4de73-c484-46fb-9585-b5156799289c None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:13.270178 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c4de73-c484-46fb-9585-b5156799289c None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:13.270655 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c4de73-c484-46fb-9585-b5156799289c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:13.273339 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515f41c5-9194-4a47-b5f1-818206f8c344 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:13.275408 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515f41c5-9194-4a47-b5f1-818206f8c344 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:13.285357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c4de73-c484-46fb-9585-b5156799289c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:13.287834 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c4de73-c484-46fb-9585-b5156799289c None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:13.291485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-515f41c5-9194-4a47-b5f1-818206f8c344 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:13.291485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:13.291485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:13.291485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:13.291485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:13.291485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:13.291485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:13.291485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:13.291485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:13.291485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:13.291485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:13.291485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:13.291485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:34:13.291485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:34:13.291485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:34:13.292777 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:34:13.292777 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:13.292777 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:13.292777 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:13.292777 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:13.292777 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:13.292777 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:13.292777 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:13.292777 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:13.292777 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:13.292777 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:13.292777 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:13.292777 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:13.292777 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:13.292777 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:13.292777 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:13.292777 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:13.293667 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:34:13.293667 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:34:13.293667 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:34:13.293667 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:34:13.293667 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:34:13.293667 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:34:13.293667 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:13.293667 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:13.293667 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 377/723] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:34:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:13.298858 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515f41c5-9194-4a47-b5f1-818206f8c344 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:13.312455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-86c4de73-c484-46fb-9585-b5156799289c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:13.312455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:13.312455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:13.312455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:13.312455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:13.312455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:13.312455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:13.312455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:13.312455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:13.312455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:13.312455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:13.312455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:13.312455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:13.312455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:13.312455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:13.312455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:13.313709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:13.313709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:13.313709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:13.313709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:13.313709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:13.313709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:13.313709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:13.313709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:13.313709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:13.313709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:13.313709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:13.313709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:13.313709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:13.313709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:13.313709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:13.313709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:13.313709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:13.314605 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:13.314605 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:13.314605 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:13.316751 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 347/724] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:13.323440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c4de73-c484-46fb-9585-b5156799289c None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:13.436029 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:13.453271 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:13.467922 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515f41c5-9194-4a47-b5f1-818206f8c344 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:13.486259 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c4de73-c484-46fb-9585-b5156799289c None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:13.506505 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1d156e-7e7c-4d33-b29c-135841ef5de6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:13.507291 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1d156e-7e7c-4d33-b29c-135841ef5de6 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:13.507699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1d156e-7e7c-4d33-b29c-135841ef5de6 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:13.508077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1d156e-7e7c-4d33-b29c-135841ef5de6 None admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:13.508888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef1d156e-7e7c-4d33-b29c-135841ef5de6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:13.509300 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef1d156e-7e7c-4d33-b29c-135841ef5de6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:13.510978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef1d156e-7e7c-4d33-b29c-135841ef5de6 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:13.525640 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 378/725] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:34:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:13.526564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b87607-8aee-41c6-8c1f-2611460ce3fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:13.526828 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1d156e-7e7c-4d33-b29c-135841ef5de6 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:13.527486 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b87607-8aee-41c6-8c1f-2611460ce3fb None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:13.527947 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b87607-8aee-41c6-8c1f-2611460ce3fb None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:13.528388 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b87607-8aee-41c6-8c1f-2611460ce3fb None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:13.529181 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b87607-8aee-41c6-8c1f-2611460ce3fb None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:13.529848 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b87607-8aee-41c6-8c1f-2611460ce3fb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:13.532460 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b87607-8aee-41c6-8c1f-2611460ce3fb None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:13.548357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 348/726] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:13 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:13.549906 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b87607-8aee-41c6-8c1f-2611460ce3fb None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:13.645618 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1d156e-7e7c-4d33-b29c-135841ef5de6 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:13.659577 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b87607-8aee-41c6-8c1f-2611460ce3fb None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:13.686748 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:13.696304 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a56ece5-6fb1-438a-8512-92c41db3389f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:13.697054 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a56ece5-6fb1-438a-8512-92c41db3389f None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:13.697407 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a56ece5-6fb1-438a-8512-92c41db3389f None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:13.697749 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a56ece5-6fb1-438a-8512-92c41db3389f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:13.713981 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a56ece5-6fb1-438a-8512-92c41db3389f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:34:13.717671 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127c288e-67fc-4150-9153-5ea9cc6fd400 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:13.718642 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127c288e-67fc-4150-9153-5ea9cc6fd400 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:13.719142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127c288e-67fc-4150-9153-5ea9cc6fd400 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:13.719588 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127c288e-67fc-4150-9153-5ea9cc6fd400 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:13.720555 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-127c288e-67fc-4150-9153-5ea9cc6fd400 None admin] RBAC: Adding query filter params (role.id=4d8fbbaea81a4145b89af12350bb067c, scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5, user.id=e0d544379fe44438ad10289b45124374) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:13.721046 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-127c288e-67fc-4150-9153-5ea9cc6fd400 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:13.721945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-4a56ece5-6fb1-438a-8512-92c41db3389f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:34:13.721945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:13.721945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:13.721945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:13.721945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:13.721945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:13.721945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:13.721945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:13.721945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:13.721945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:13.721945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:13.721945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:13.721945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:34:13.721945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:34:13.721945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:34:13.721945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:34:13.722824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:34:13.722824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:34:13.722824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:34:13.722824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:34:13.722824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:13.722824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:13.722824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:13.722824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:13.722824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:13.722824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:13.722824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:13.722824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:13.722824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:13.722824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:13.722824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:13.722824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:13.724719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:13.724719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:13.724719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:13.724719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:13.724719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:34:13.724719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:34:13.724719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:34:13.724719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:34:13.724719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:34:13.724719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:34:13.724719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:34:13.724719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:13.727077 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-127c288e-67fc-4150-9153-5ea9cc6fd400 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:13.728190 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 379/727] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:34:13 2022] GET /identity/v3/roles/admin => generated 83 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:13.729169 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a56ece5-6fb1-438a-8512-92c41db3389f None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:13.738127 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 349/728] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:34:13 2022] GET /identity/v3/role_assignments?user.id=e0d544379fe44438ad10289b45124374&scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5&role.id=4d8fbbaea81a4145b89af12350bb067c => generated 260 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:13.739432 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127c288e-67fc-4150-9153-5ea9cc6fd400 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:13.747443 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:13.924452 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a56ece5-6fb1-438a-8512-92c41db3389f None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:13.950749 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127c288e-67fc-4150-9153-5ea9cc6fd400 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:13.972919 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bdfe47b-581d-4438-bbc9-458592a7f083 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:13.973605 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdfe47b-581d-4438-bbc9-458592a7f083 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:13.973960 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdfe47b-581d-4438-bbc9-458592a7f083 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:13.974339 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdfe47b-581d-4438-bbc9-458592a7f083 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:13.985135 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bdfe47b-581d-4438-bbc9-458592a7f083 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:13.986756 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bdfe47b-581d-4438-bbc9-458592a7f083 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:14.000177 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-8bdfe47b-581d-4438-bbc9-458592a7f083 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 24 15:34:14.000177 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:14.000177 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:14.000177 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:14.000177 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:14.000177 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:14.000177 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:14.000177 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:14.000177 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:14.000177 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:14.000177 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:14.000177 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:14.000177 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:34:14.000177 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:34:14.000177 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:34:14.000177 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:34:14.001193 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:14.001193 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:14.001193 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:34:14.001193 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:14.001193 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:34:14.001193 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:14.001193 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:14.001193 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:14.001193 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:14.001193 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:14.001193 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:14.001193 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:14.001193 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:14.001193 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:14.001193 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:14.001193 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:14.001193 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:14.001193 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:14.002117 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:14.002117 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:14.002117 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:34:14.002117 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:34:14.002117 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:34:14.002117 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:34:14.002117 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:34:14.002117 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:34:14.002117 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 24 15:34:14.002117 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:14.005247 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcac927c-6aa6-4c7c-af52-315161f04cb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:14.006078 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcac927c-6aa6-4c7c-af52-315161f04cb3 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:14.006306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 380/729] 10.222.0.102 () {56 vars in 1122 bytes} [Wed Aug 24 15:34:13 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 273 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:14.006718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcac927c-6aa6-4c7c-af52-315161f04cb3 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:14.007165 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcac927c-6aa6-4c7c-af52-315161f04cb3 None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:14.007840 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398edfd6-8150-490d-88af-31d6c1fde1d5 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:14.008216 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398edfd6-8150-490d-88af-31d6c1fde1d5 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:14.008429 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcac927c-6aa6-4c7c-af52-315161f04cb3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:14.008580 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398edfd6-8150-490d-88af-31d6c1fde1d5 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:14.009046 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcac927c-6aa6-4c7c-af52-315161f04cb3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:14.014782 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 381/730] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:14 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:14.014782 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398edfd6-8150-490d-88af-31d6c1fde1d5 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:14.016058 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcac927c-6aa6-4c7c-af52-315161f04cb3 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:14.029238 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 350/731] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:34:13 2022] GET /identity/v3/roles?name=admin => generated 341 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:14.031791 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c4e4b7-7c06-4c0a-a9c0-181231168b95 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:14.032824 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c4e4b7-7c06-4c0a-a9c0-181231168b95 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:14.033455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c4e4b7-7c06-4c0a-a9c0-181231168b95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:14.055451 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-61c4e4b7-7c06-4c0a-a9c0-181231168b95 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:14.057013 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-61c4e4b7-7c06-4c0a-a9c0-181231168b95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:14.090505 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:14.104994 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:14.132381 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398edfd6-8150-490d-88af-31d6c1fde1d5 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:14.137300 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 351/732] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:14 2022] POST /identity/v3/auth/tokens => generated 800 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:34:14.138494 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c4e4b7-7c06-4c0a-a9c0-181231168b95 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:14.205803 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23245c72-44f4-4941-a596-0e4eb867767c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:14.206528 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23245c72-44f4-4941-a596-0e4eb867767c None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:14.206741 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23245c72-44f4-4941-a596-0e4eb867767c None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:14.207018 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23245c72-44f4-4941-a596-0e4eb867767c None admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:14.207868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23245c72-44f4-4941-a596-0e4eb867767c None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:14.208100 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23245c72-44f4-4941-a596-0e4eb867767c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:14.209570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23245c72-44f4-4941-a596-0e4eb867767c None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:14.247882 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 382/733] 10.222.0.102 () {56 vars in 1154 bytes} [Wed Aug 24 15:34:14 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 386 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:14.254660 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23245c72-44f4-4941-a596-0e4eb867767c None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:14.257457 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c4e4b7-7c06-4c0a-a9c0-181231168b95 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:14.330576 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b66638cc-9298-4b28-81d7-65acdc20c8c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:14.331228 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66638cc-9298-4b28-81d7-65acdc20c8c8 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:14.331557 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66638cc-9298-4b28-81d7-65acdc20c8c8 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:14.331856 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66638cc-9298-4b28-81d7-65acdc20c8c8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:14.352245 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b66638cc-9298-4b28-81d7-65acdc20c8c8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:14.353979 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b66638cc-9298-4b28-81d7-65acdc20c8c8 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:14.382916 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-b66638cc-9298-4b28-81d7-65acdc20c8c8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:14.382916 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:14.382916 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:14.382916 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:14.382916 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:14.382916 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:14.382916 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:14.382916 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:14.382916 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:14.382916 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:14.382916 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:14.382916 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:14.382916 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:14.382916 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:14.382916 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:14.382916 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:14.383988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:14.383988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:14.383988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:14.383988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:14.383988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:14.383988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:14.383988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:14.383988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:14.383988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:14.383988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:14.383988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:14.383988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:14.383988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:14.383988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:14.383988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:14.383988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:14.383988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:14.384864 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:14.384864 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:14.384864 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:14.384864 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 352/734] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:14.390654 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b66638cc-9298-4b28-81d7-65acdc20c8c8 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:14.431376 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:14.456904 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23245c72-44f4-4941-a596-0e4eb867767c None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:14.509040 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:14.530106 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f320df-0946-45af-b427-a5120416e66e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:14.530749 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f320df-0946-45af-b427-a5120416e66e None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:14.531097 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f320df-0946-45af-b427-a5120416e66e None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:14.531375 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f320df-0946-45af-b427-a5120416e66e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:14.562099 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b66638cc-9298-4b28-81d7-65acdc20c8c8 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:14.599060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e014d6-c95a-4656-95ef-9310151ac993 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:14.599691 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e014d6-c95a-4656-95ef-9310151ac993 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:14.599977 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e014d6-c95a-4656-95ef-9310151ac993 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:14.600253 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e014d6-c95a-4656-95ef-9310151ac993 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:14.600900 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e014d6-c95a-4656-95ef-9310151ac993 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:14.601797 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e014d6-c95a-4656-95ef-9310151ac993 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:14.603448 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e014d6-c95a-4656-95ef-9310151ac993 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:14.620710 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 353/735] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:14 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:14.620999 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f320df-0946-45af-b427-a5120416e66e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:14.621537 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e014d6-c95a-4656-95ef-9310151ac993 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:14.622529 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f320df-0946-45af-b427-a5120416e66e None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:14.749141 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 383/736] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:34:14 2022] GET /identity/v3/auth/tokens => generated 800 bytes in 490 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 24 15:34:14.750181 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:14.752452 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f320df-0946-45af-b427-a5120416e66e None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:14.765076 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e014d6-c95a-4656-95ef-9310151ac993 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:14.785822 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:14.819296 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b97104-e936-4959-8b4a-0ad91bb228d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:14.820041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b97104-e936-4959-8b4a-0ad91bb228d0 None admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:14.820392 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b97104-e936-4959-8b4a-0ad91bb228d0 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:14.820729 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b97104-e936-4959-8b4a-0ad91bb228d0 None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:14.821935 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b97104-e936-4959-8b4a-0ad91bb228d0 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:14.823738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b97104-e936-4959-8b4a-0ad91bb228d0 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:14.851795 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 354/737] 10.222.0.102 () {60 vars in 1146 bytes} [Wed Aug 24 15:34:14 2022] POST /identity/v3/services => generated 252 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:34:14.852892 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b97104-e936-4959-8b4a-0ad91bb228d0 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:14.891617 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f320df-0946-45af-b427-a5120416e66e None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:14.940432 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f96a5ce-5f95-4871-a4cc-05245348cd8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:14.941440 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f96a5ce-5f95-4871-a4cc-05245348cd8a None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:14.941922 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f96a5ce-5f95-4871-a4cc-05245348cd8a None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:14.942409 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f96a5ce-5f95-4871-a4cc-05245348cd8a None admin] PATH_INFO: `/v3/users/swift` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:14.952344 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f96a5ce-5f95-4871-a4cc-05245348cd8a None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:14.954482 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f96a5ce-5f95-4871-a4cc-05245348cd8a None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:14.968620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-8f96a5ce-5f95-4871-a4cc-05245348cd8a None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 24 15:34:14.968620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:14.968620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:14.968620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:14.968620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:14.968620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:14.968620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:14.968620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:14.968620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:14.968620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:14.968620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:14.968620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:14.968620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:34:14.968620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:34:14.968620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:34:14.968620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:34:14.969626 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:14.969626 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:14.969626 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:34:14.969626 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:14.969626 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:34:14.969626 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:14.969626 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:14.969626 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:14.969626 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:14.969626 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:14.969626 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:14.969626 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:14.969626 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:14.969626 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:14.969626 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:14.969626 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:14.969626 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:14.969626 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:14.970596 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:14.970596 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:14.970596 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:34:14.970596 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:34:14.970596 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:34:14.970596 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:34:14.970596 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:34:14.970596 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:34:14.970596 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 24 15:34:14.970596 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:14.973341 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 384/738] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:34:14 2022] GET /identity/v3/users/swift => generated 83 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:14.979807 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f96a5ce-5f95-4871-a4cc-05245348cd8a None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:15.076251 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:15.100804 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f96a5ce-5f95-4871-a4cc-05245348cd8a None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:15.102910 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b97104-e936-4959-8b4a-0ad91bb228d0 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:15.139035 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a91e14-9764-4b39-885a-5de171bc31be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:15.140526 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a91e14-9764-4b39-885a-5de171bc31be None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:15.141030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a91e14-9764-4b39-885a-5de171bc31be None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:15.141498 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a91e14-9764-4b39-885a-5de171bc31be None admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:15.142429 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a91e14-9764-4b39-885a-5de171bc31be None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:15.142955 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a91e14-9764-4b39-885a-5de171bc31be None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:15.144680 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a91e14-9764-4b39-885a-5de171bc31be None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:15.166961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 385/739] 10.222.0.102 () {56 vars in 1140 bytes} [Wed Aug 24 15:34:14 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 372 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:15.171689 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e4ad5c-34a4-4bbe-a35e-3863a629939a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:15.171689 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e4ad5c-34a4-4bbe-a35e-3863a629939a None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:15.171689 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e4ad5c-34a4-4bbe-a35e-3863a629939a None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:15.171689 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e4ad5c-34a4-4bbe-a35e-3863a629939a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:15.173917 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a91e14-9764-4b39-885a-5de171bc31be None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:15.184078 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e4ad5c-34a4-4bbe-a35e-3863a629939a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:15.186980 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e4ad5c-34a4-4bbe-a35e-3863a629939a None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:15.207895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-b9e4ad5c-34a4-4bbe-a35e-3863a629939a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:15.207895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:15.207895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:15.207895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:15.207895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:15.207895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:15.207895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:15.207895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:15.207895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:15.207895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:15.207895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:15.207895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:15.207895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:15.207895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:15.207895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:15.207895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:15.208951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:15.208951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:15.208951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:15.208951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:15.208951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:15.208951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:15.208951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:15.208951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:15.208951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:15.208951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:15.208951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:15.208951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:15.208951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:15.208951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:15.208951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:15.208951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:15.208951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:15.209845 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:15.209845 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:15.209845 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:15.211980 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:15.212399 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 355/740] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 360 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:15.218242 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e4ad5c-34a4-4bbe-a35e-3863a629939a None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:15.307986 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a91e14-9764-4b39-885a-5de171bc31be None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:15.354980 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffdeafb-fe03-4ded-8ef7-0fc9caf81db6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:15.355681 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffdeafb-fe03-4ded-8ef7-0fc9caf81db6 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:15.356073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e4ad5c-34a4-4bbe-a35e-3863a629939a None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:15.356192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffdeafb-fe03-4ded-8ef7-0fc9caf81db6 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:15.356581 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffdeafb-fe03-4ded-8ef7-0fc9caf81db6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:15.388565 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:15.402247 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc09ebbb-0e29-4fae-a9d2-a8ed60cf30e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:15.403229 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc09ebbb-0e29-4fae-a9d2-a8ed60cf30e8 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:15.403761 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc09ebbb-0e29-4fae-a9d2-a8ed60cf30e8 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:15.404242 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc09ebbb-0e29-4fae-a9d2-a8ed60cf30e8 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:15.405981 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc09ebbb-0e29-4fae-a9d2-a8ed60cf30e8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:15.406525 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc09ebbb-0e29-4fae-a9d2-a8ed60cf30e8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:15.408290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc09ebbb-0e29-4fae-a9d2-a8ed60cf30e8 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:15.422558 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 356/741] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:15 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:15.424275 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0617c384-51a5-4dc6-8f7b-e6ff278008a6 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:15.424786 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0617c384-51a5-4dc6-8f7b-e6ff278008a6 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:15.425280 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0617c384-51a5-4dc6-8f7b-e6ff278008a6 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:15.431350 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 357/742] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:15 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:15.432552 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0617c384-51a5-4dc6-8f7b-e6ff278008a6 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:15.443972 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ffdeafb-fe03-4ded-8ef7-0fc9caf81db6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:15.445459 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ffdeafb-fe03-4ded-8ef7-0fc9caf81db6 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:15.510030 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:15.562295 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 386/743] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:34:15 2022] GET /identity/v3/auth/tokens => generated 904 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 24 15:34:15.563981 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6a7a96-3775-4351-9aaf-0e17f0c56c7b None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:15.564745 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6a7a96-3775-4351-9aaf-0e17f0c56c7b None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:15.565165 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6a7a96-3775-4351-9aaf-0e17f0c56c7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:15.570216 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0617c384-51a5-4dc6-8f7b-e6ff278008a6 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:15.585147 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-9b6a7a96-3775-4351-9aaf-0e17f0c56c7b None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:15.586381 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-9b6a7a96-3775-4351-9aaf-0e17f0c56c7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:15.616003 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301be6df-f2d9-48ae-beab-ad6bb12f458f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:15.617224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301be6df-f2d9-48ae-beab-ad6bb12f458f None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:15.618131 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301be6df-f2d9-48ae-beab-ad6bb12f458f None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:15.618880 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301be6df-f2d9-48ae-beab-ad6bb12f458f None admin] PATH_INFO: `/v3/projects/service` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:15.626766 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:15.628853 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301be6df-f2d9-48ae-beab-ad6bb12f458f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:15.631590 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301be6df-f2d9-48ae-beab-ad6bb12f458f None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:15.649932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-301be6df-f2d9-48ae-beab-ad6bb12f458f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:15.649932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:15.649932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:15.649932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:15.649932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:15.649932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:15.649932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:15.649932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:15.649932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:15.649932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:15.649932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:15.649932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:15.649932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:34:15.649932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:34:15.649932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:34:15.650902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:34:15.650902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:15.650902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:15.650902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:15.650902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:15.650902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:15.650902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:15.650902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:15.650902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:15.650902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:15.650902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:15.650902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:15.650902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:15.650902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:15.650902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:15.650902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:15.650902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:15.651836 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:34:15.651836 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:34:15.651836 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:34:15.651836 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:34:15.651836 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:34:15.651836 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:34:15.651836 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:15.651836 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:15.655784 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 358/744] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:34:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:15.656885 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301be6df-f2d9-48ae-beab-ad6bb12f458f None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:15.680379 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 387/745] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:15 2022] POST /identity/v3/auth/tokens => generated 904 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:34:15.681919 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6a7a96-3775-4351-9aaf-0e17f0c56c7b None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:15.792092 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301be6df-f2d9-48ae-beab-ad6bb12f458f None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:15.814399 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:15.840345 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60df81f3-73fd-424f-aed4-f30d8173b3a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:15.842668 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60df81f3-73fd-424f-aed4-f30d8173b3a8 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:15.844533 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60df81f3-73fd-424f-aed4-f30d8173b3a8 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:15.845029 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60df81f3-73fd-424f-aed4-f30d8173b3a8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:15.872452 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60df81f3-73fd-424f-aed4-f30d8173b3a8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:15.874944 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60df81f3-73fd-424f-aed4-f30d8173b3a8 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:15.898282 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-60df81f3-73fd-424f-aed4-f30d8173b3a8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:15.898282 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:15.898282 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:15.898282 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:15.898282 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:15.898282 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:15.898282 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:15.898282 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:15.898282 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:15.898282 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:15.898282 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:15.898282 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:15.898282 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:15.898282 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:15.898282 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:15.898282 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:15.899353 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:15.899353 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:15.899353 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:15.899353 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:15.899353 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:15.899353 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:15.899353 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:15.899353 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:15.899353 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:15.899353 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:15.899353 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:15.899353 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:15.899353 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:15.899353 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:15.899353 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:15.899353 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:15.899353 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:15.900277 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:15.900277 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:15.900277 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:15.900277 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 359/746] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:15.900919 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cd1d74-f268-4284-ac95-7cb2c358ff50 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:15.903897 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cd1d74-f268-4284-ac95-7cb2c358ff50 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:15.903897 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cd1d74-f268-4284-ac95-7cb2c358ff50 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:15.903897 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 360/747] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:15 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:15.908237 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cd1d74-f268-4284-ac95-7cb2c358ff50 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:15.918478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6a7a96-3775-4351-9aaf-0e17f0c56c7b None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:15.955469 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb6cf1a-2146-434f-8afe-e5c9c1f7c95e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:15.956168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb6cf1a-2146-434f-8afe-e5c9c1f7c95e None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:15.956445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb6cf1a-2146-434f-8afe-e5c9c1f7c95e None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:15.956713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb6cf1a-2146-434f-8afe-e5c9c1f7c95e None admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:15.957453 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb6cf1a-2146-434f-8afe-e5c9c1f7c95e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:15.957759 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb6cf1a-2146-434f-8afe-e5c9c1f7c95e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:15.959187 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb6cf1a-2146-434f-8afe-e5c9c1f7c95e None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:15.971815 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 388/748] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:34:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:15.973072 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4425680-bace-4bc9-8efd-37ecd419c441 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:15.973354 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4425680-bace-4bc9-8efd-37ecd419c441 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:15.973683 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4425680-bace-4bc9-8efd-37ecd419c441 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:15.992503 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-b4425680-bace-4bc9-8efd-37ecd419c441 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:15.993700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-b4425680-bace-4bc9-8efd-37ecd419c441 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:16.021724 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:16.024850 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cd1d74-f268-4284-ac95-7cb2c358ff50 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:16.069371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646477dd-7abc-4f43-a0e7-e80427419a18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:16.069993 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646477dd-7abc-4f43-a0e7-e80427419a18 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:16.070275 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646477dd-7abc-4f43-a0e7-e80427419a18 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:16.070553 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646477dd-7abc-4f43-a0e7-e80427419a18 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:16.071668 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-646477dd-7abc-4f43-a0e7-e80427419a18 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:16.071668 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-646477dd-7abc-4f43-a0e7-e80427419a18 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:16.072917 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-646477dd-7abc-4f43-a0e7-e80427419a18 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:16.084445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 389/749] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:15 2022] POST /identity/v3/auth/tokens => generated 904 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 15:34:16.085549 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4425680-bace-4bc9-8efd-37ecd419c441 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:16.086452 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 361/750] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:15 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:16.087341 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646477dd-7abc-4f43-a0e7-e80427419a18 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:16.134691 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:16.216543 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646477dd-7abc-4f43-a0e7-e80427419a18 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:16.225362 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4425680-bace-4bc9-8efd-37ecd419c441 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:16.255206 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6495a0-a4cf-44b2-a271-063fae428043 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:16.256237 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6495a0-a4cf-44b2-a271-063fae428043 None admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:16.256783 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6495a0-a4cf-44b2-a271-063fae428043 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:16.257250 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6495a0-a4cf-44b2-a271-063fae428043 None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:16.258133 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a6495a0-a4cf-44b2-a271-063fae428043 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:16.259740 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a6495a0-a4cf-44b2-a271-063fae428043 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:16.272888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698eb3cc-457d-4fb0-be12-e8b538b40bdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:16.273934 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698eb3cc-457d-4fb0-be12-e8b538b40bdd None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:16.274358 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698eb3cc-457d-4fb0-be12-e8b538b40bdd None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:16.274771 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698eb3cc-457d-4fb0-be12-e8b538b40bdd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:16.276154 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-698eb3cc-457d-4fb0-be12-e8b538b40bdd None admin] RBAC: Adding query filter params (role.id=628bab09d1dc48fd9e01023b2f7671f6, scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5, user.id=a6c17620ead14cb49d962990e1b1e0d8) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:16.276634 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-698eb3cc-457d-4fb0-be12-e8b538b40bdd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:16.278889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-698eb3cc-457d-4fb0-be12-e8b538b40bdd None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:16.279415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 362/751] 10.222.0.102 () {60 vars in 1146 bytes} [Wed Aug 24 15:34:16 2022] POST /identity/v3/services => generated 241 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:34:16.281140 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6495a0-a4cf-44b2-a271-063fae428043 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:16.292047 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 390/752] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:34:16 2022] GET /identity/v3/role_assignments?user.id=a6c17620ead14cb49d962990e1b1e0d8&scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5&role.id=628bab09d1dc48fd9e01023b2f7671f6 => generated 260 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:16.394419 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:16.423594 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6495a0-a4cf-44b2-a271-063fae428043 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:16.445491 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698eb3cc-457d-4fb0-be12-e8b538b40bdd None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:16.475396 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c302ecf-be90-4d84-b07f-8146d7259be2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:16.476270 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c302ecf-be90-4d84-b07f-8146d7259be2 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:16.476551 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c302ecf-be90-4d84-b07f-8146d7259be2 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:16.477188 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c302ecf-be90-4d84-b07f-8146d7259be2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:16.484988 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:16.486860 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c302ecf-be90-4d84-b07f-8146d7259be2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:16.488685 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c302ecf-be90-4d84-b07f-8146d7259be2 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:16.508716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-8c302ecf-be90-4d84-b07f-8146d7259be2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:16.508716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:16.508716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:16.508716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:16.508716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:16.508716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:16.508716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:16.508716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:16.508716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:16.508716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:16.508716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:16.508716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:16.508716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:34:16.508716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:34:16.508716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:34:16.509853 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:34:16.509853 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:16.509853 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:16.509853 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:16.509853 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:16.509853 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:16.509853 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:16.509853 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:16.509853 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:16.509853 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:16.509853 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:16.509853 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:16.509853 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:16.509853 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:16.509853 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:16.509853 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:16.509853 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:16.510926 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:34:16.510926 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:34:16.510926 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:34:16.510926 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:34:16.510926 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:34:16.510926 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:34:16.510926 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:16.510926 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:16.510926 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 363/753] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:34:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:16.520188 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c302ecf-be90-4d84-b07f-8146d7259be2 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:16.623963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698eb3cc-457d-4fb0-be12-e8b538b40bdd None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:16.666111 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d463fae-a5dc-4c04-bd78-308ae8a288d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:16.666979 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d463fae-a5dc-4c04-bd78-308ae8a288d9 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:16.667271 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d463fae-a5dc-4c04-bd78-308ae8a288d9 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:16.667620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d463fae-a5dc-4c04-bd78-308ae8a288d9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:16.682285 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d463fae-a5dc-4c04-bd78-308ae8a288d9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:34:16.688899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-6d463fae-a5dc-4c04-bd78-308ae8a288d9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:34:16.688899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:16.688899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:16.688899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:16.688899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:16.688899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:16.688899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:16.688899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:16.688899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:16.688899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:16.688899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:16.688899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:16.688899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:34:16.688899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:34:16.688899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:34:16.688899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:34:16.691113 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:34:16.691113 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:34:16.691113 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:34:16.691113 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:34:16.691113 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:16.691113 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:16.691113 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:16.691113 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:16.691113 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:16.691113 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:16.691113 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:16.691113 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:16.691113 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:16.691113 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:16.691113 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:16.691113 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:16.693046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:16.693046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:16.693046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:16.693046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:16.693046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:34:16.693046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:34:16.693046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:34:16.693046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:34:16.693046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:34:16.693046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:34:16.693046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:34:16.693046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:16.693046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 391/754] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:34:16 2022] GET /identity/v3/roles/admin => generated 83 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:34:16.694659 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c302ecf-be90-4d84-b07f-8146d7259be2 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:16.698727 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d463fae-a5dc-4c04-bd78-308ae8a288d9 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:16.748262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac7bd2a-a54c-4b44-b238-e32b9cfdb830 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:16.750963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac7bd2a-a54c-4b44-b238-e32b9cfdb830 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:16.751440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac7bd2a-a54c-4b44-b238-e32b9cfdb830 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:16.751941 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac7bd2a-a54c-4b44-b238-e32b9cfdb830 None admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:16.753422 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ac7bd2a-a54c-4b44-b238-e32b9cfdb830 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:16.753949 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ac7bd2a-a54c-4b44-b238-e32b9cfdb830 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:16.756682 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ac7bd2a-a54c-4b44-b238-e32b9cfdb830 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:16.782783 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 364/755] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:34:16 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:16.789826 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac7bd2a-a54c-4b44-b238-e32b9cfdb830 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:16.804140 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:16.870702 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:16.894724 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d463fae-a5dc-4c04-bd78-308ae8a288d9 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:16.996492 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60948d20-5715-4fa0-8776-54c661683616 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:16.997046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60948d20-5715-4fa0-8776-54c661683616 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:16.997322 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60948d20-5715-4fa0-8776-54c661683616 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:16.997595 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60948d20-5715-4fa0-8776-54c661683616 None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:16.998315 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60948d20-5715-4fa0-8776-54c661683616 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:16.998651 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60948d20-5715-4fa0-8776-54c661683616 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:17.000074 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60948d20-5715-4fa0-8776-54c661683616 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:17.012437 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 392/756] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:34:16 2022] GET /identity/v3/roles?name=admin => generated 341 bytes in 314 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:17.018604 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60948d20-5715-4fa0-8776-54c661683616 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:17.067076 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac7bd2a-a54c-4b44-b238-e32b9cfdb830 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:17.106922 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813a5161-df21-4380-8c69-4ae6a8d54059 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:17.107583 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813a5161-df21-4380-8c69-4ae6a8d54059 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:17.108556 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813a5161-df21-4380-8c69-4ae6a8d54059 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:17.109601 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813a5161-df21-4380-8c69-4ae6a8d54059 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:17.110371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-813a5161-df21-4380-8c69-4ae6a8d54059 None admin] RBAC: Adding query filter params (role.id=4d8fbbaea81a4145b89af12350bb067c, scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5, user.id=6a070d4fd8c4466c857bd03965c11349) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:17.110686 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-813a5161-df21-4380-8c69-4ae6a8d54059 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:17.112203 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-813a5161-df21-4380-8c69-4ae6a8d54059 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:17.122930 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 365/757] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:34:16 2022] GET /identity/v3/role_assignments?user.id=6a070d4fd8c4466c857bd03965c11349&scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5&role.id=4d8fbbaea81a4145b89af12350bb067c => generated 260 bytes in 334 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:17.180381 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60948d20-5715-4fa0-8776-54c661683616 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:17.228364 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7316c7d-540a-4320-b9f9-7f8eee764029 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:17.230433 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7316c7d-540a-4320-b9f9-7f8eee764029 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:17.230433 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7316c7d-540a-4320-b9f9-7f8eee764029 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:17.230433 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 366/758] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:17 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:34:17.234364 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69aff01f-8b07-484d-8392-e902fed527af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:17.234364 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69aff01f-8b07-484d-8392-e902fed527af None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:17.234364 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69aff01f-8b07-484d-8392-e902fed527af None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:17.234364 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69aff01f-8b07-484d-8392-e902fed527af None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:17.236340 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2adf0529-ab78-4b36-ad11-9629d1c01a33 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:17.236621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2adf0529-ab78-4b36-ad11-9629d1c01a33 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:17.236958 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2adf0529-ab78-4b36-ad11-9629d1c01a33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:17.268201 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69aff01f-8b07-484d-8392-e902fed527af None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:17.269651 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69aff01f-8b07-484d-8392-e902fed527af None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:17.275681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-2adf0529-ab78-4b36-ad11-9629d1c01a33 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:17.276931 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-2adf0529-ab78-4b36-ad11-9629d1c01a33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:17.287352 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:17.305889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-69aff01f-8b07-484d-8392-e902fed527af None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:17.305889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:17.305889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:17.305889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:17.305889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:17.305889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:17.305889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:17.305889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:17.305889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:17.305889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:17.305889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:17.305889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:17.305889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:17.305889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:17.305889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:17.305889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:17.308466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:17.308466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:17.308466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:17.308466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:17.308466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:17.308466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:17.308466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:17.308466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:17.308466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:17.308466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:17.308466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:17.308466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:17.308466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:17.308466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:17.308466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:17.308466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:17.308466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:17.310551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:17.310551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:17.310551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:17.310551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 393/759] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:17.318528 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69aff01f-8b07-484d-8392-e902fed527af None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:17.392470 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:17.399248 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 367/760] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:17 2022] POST /identity/v3/auth/tokens => generated 1003 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 15:34:17.525038 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69aff01f-8b07-484d-8392-e902fed527af None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:17.566026 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf5572e-e6c6-4473-a986-7a69473a0dd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:17.566980 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf5572e-e6c6-4473-a986-7a69473a0dd2 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:17.567271 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf5572e-e6c6-4473-a986-7a69473a0dd2 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:17.567723 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf5572e-e6c6-4473-a986-7a69473a0dd2 None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:17.568961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf5572e-e6c6-4473-a986-7a69473a0dd2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:17.569490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf5572e-e6c6-4473-a986-7a69473a0dd2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:17.571115 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf5572e-e6c6-4473-a986-7a69473a0dd2 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:17.590990 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 394/761] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:17 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:17.597055 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2adf0529-ab78-4b36-ad11-9629d1c01a33 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:17.721423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2adf0529-ab78-4b36-ad11-9629d1c01a33 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:17.762739 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2125f4-3320-4ccc-ae48-0515d9e1ae86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:17.763384 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2125f4-3320-4ccc-ae48-0515d9e1ae86 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:17.763680 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2125f4-3320-4ccc-ae48-0515d9e1ae86 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:17.763957 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2125f4-3320-4ccc-ae48-0515d9e1ae86 None admin] PATH_INFO: `/v3/users/nova` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:17.776902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2125f4-3320-4ccc-ae48-0515d9e1ae86 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:17.778300 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2125f4-3320-4ccc-ae48-0515d9e1ae86 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:17.797440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-1b2125f4-3320-4ccc-ae48-0515d9e1ae86 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 24 15:34:17.797440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:17.797440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:17.797440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:17.797440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:17.797440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:17.797440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:17.797440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:17.797440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:17.797440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:17.797440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:17.797440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:17.797440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:34:17.797440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:34:17.797440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:34:17.797440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:34:17.799691 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:17.799691 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:17.799691 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:34:17.799691 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:17.799691 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:34:17.799691 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:17.799691 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:17.799691 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:17.799691 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:17.799691 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:17.799691 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:17.799691 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:17.799691 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:17.799691 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:17.799691 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:17.799691 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:17.799691 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:17.799691 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:17.802055 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:17.802055 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:17.802055 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:34:17.802055 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:34:17.802055 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:34:17.802055 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:34:17.802055 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:34:17.802055 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:34:17.802055 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 24 15:34:17.802055 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:17.802055 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 368/762] 10.222.0.102 () {56 vars in 1098 bytes} [Wed Aug 24 15:34:17 2022] GET /identity/v3/users/nova => generated 82 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:17.809184 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf5572e-e6c6-4473-a986-7a69473a0dd2 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:17.846759 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2125f4-3320-4ccc-ae48-0515d9e1ae86 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:17.913842 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:17.972248 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2125f4-3320-4ccc-ae48-0515d9e1ae86 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:18.010733 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:18.028229 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ed80c2-0a47-47c3-b569-3739cb8753e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:18.028781 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ed80c2-0a47-47c3-b569-3739cb8753e6 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:18.029058 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ed80c2-0a47-47c3-b569-3739cb8753e6 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:18.029328 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ed80c2-0a47-47c3-b569-3739cb8753e6 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:18.032058 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ed80c2-0a47-47c3-b569-3739cb8753e6 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:18.033803 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ed80c2-0a47-47c3-b569-3739cb8753e6 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:18.053286 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-d4ed80c2-0a47-47c3-b569-3739cb8753e6 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 24 15:34:18.053286 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:18.053286 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 15:34:18.053286 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 15:34:18.053286 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 15:34:18.053286 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 15:34:18.053286 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 15:34:18.053286 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:18.053286 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 24 15:34:18.053286 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:18.053286 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 15:34:18.053286 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:18.053286 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:18.053286 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:18.053286 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:18.053286 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:18.053286 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:18.054333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:18.054333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:18.054333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:18.054333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:18.054333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:18.054333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:18.054333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 15:34:18.054333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 15:34:18.054333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 15:34:18.054333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 15:34:18.054333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:18.054333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:18.054333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:18.054333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:18.054333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:18.054333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:18.055148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:18.055148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:18.055148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:18.055148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:18.055148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:18.055148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:18.055148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:18.055148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:18.055148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:18.055148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:18.055148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 15:34:18.055148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:18.055148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 24 15:34:18.055148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:18.057978 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 369/763] 10.222.0.102 () {56 vars in 1134 bytes} [Wed Aug 24 15:34:17 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:34:18.073851 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf5572e-e6c6-4473-a986-7a69473a0dd2 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:18.078694 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ed80c2-0a47-47c3-b569-3739cb8753e6 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:18.112748 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1e8b79-939e-4f0e-a4e6-aca04521d942 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:18.113508 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1e8b79-939e-4f0e-a4e6-aca04521d942 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:18.113674 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1e8b79-939e-4f0e-a4e6-aca04521d942 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:18.113882 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1e8b79-939e-4f0e-a4e6-aca04521d942 None admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:18.114683 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1e8b79-939e-4f0e-a4e6-aca04521d942 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:18.114990 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1e8b79-939e-4f0e-a4e6-aca04521d942 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:18.123937 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1e8b79-939e-4f0e-a4e6-aca04521d942 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:18.139752 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 395/764] 10.222.0.102 () {56 vars in 1138 bytes} [Wed Aug 24 15:34:17 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 370 bytes in 327 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:18.142422 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1e8b79-939e-4f0e-a4e6-aca04521d942 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:18.263216 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1e8b79-939e-4f0e-a4e6-aca04521d942 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:18.283368 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:18.300616 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ed80c2-0a47-47c3-b569-3739cb8753e6 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:18.309274 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d547d0-7525-4618-8a40-2407bc6c1860 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:18.309920 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d547d0-7525-4618-8a40-2407bc6c1860 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:18.310205 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d547d0-7525-4618-8a40-2407bc6c1860 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:18.310482 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d547d0-7525-4618-8a40-2407bc6c1860 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:18.341741 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d547d0-7525-4618-8a40-2407bc6c1860 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:18.343279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d547d0-7525-4618-8a40-2407bc6c1860 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:18.354489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8153de2-4328-4695-8703-232a2d4faf25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:18.355109 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8153de2-4328-4695-8703-232a2d4faf25 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:18.355396 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8153de2-4328-4695-8703-232a2d4faf25 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:18.355703 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8153de2-4328-4695-8703-232a2d4faf25 None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:18.356364 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8153de2-4328-4695-8703-232a2d4faf25 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:18.356676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8153de2-4328-4695-8703-232a2d4faf25 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:18.357134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:18.363959 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8153de2-4328-4695-8703-232a2d4faf25 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:18.375700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-c9d547d0-7525-4618-8a40-2407bc6c1860 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:18.375700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:18.375700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:18.375700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:18.375700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:18.375700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:18.375700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:18.375700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:18.375700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:18.375700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:18.375700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:18.375700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:18.375700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:18.375700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:18.375700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:18.375700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:18.376844 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:18.376844 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:18.376844 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:18.376844 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:18.376844 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:18.376844 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:18.376844 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:18.376844 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:18.376844 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:18.376844 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:18.376844 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:18.376844 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:18.376844 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:18.376844 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:18.376844 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:18.376844 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:18.376844 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:18.377720 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:18.377720 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:18.377720 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:18.377720 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 396/765] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:18.383695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 370/766] 10.222.0.102 () {56 vars in 1129 bytes} [Wed Aug 24 15:34:18 2022] GET /identity/v3/services?name=block-storage => generated 131 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:18.384983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d547d0-7525-4618-8a40-2407bc6c1860 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:18.385105 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8153de2-4328-4695-8703-232a2d4faf25 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:18.561023 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d547d0-7525-4618-8a40-2407bc6c1860 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:18.582232 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8153de2-4328-4695-8703-232a2d4faf25 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:18.605095 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74357d3-43d2-41b3-a3e1-8ce3626704a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:18.605769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74357d3-43d2-41b3-a3e1-8ce3626704a0 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:18.606054 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74357d3-43d2-41b3-a3e1-8ce3626704a0 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:18.606320 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74357d3-43d2-41b3-a3e1-8ce3626704a0 None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:18.606967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74357d3-43d2-41b3-a3e1-8ce3626704a0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:18.607268 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74357d3-43d2-41b3-a3e1-8ce3626704a0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:18.609271 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74357d3-43d2-41b3-a3e1-8ce3626704a0 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:18.637201 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 397/767] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:18 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:18.643553 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74357d3-43d2-41b3-a3e1-8ce3626704a0 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:18.648450 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa0713b-a256-438d-8ef3-6415b63b344e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:18.649056 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa0713b-a256-438d-8ef3-6415b63b344e None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:18.649339 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa0713b-a256-438d-8ef3-6415b63b344e None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:18.649608 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa0713b-a256-438d-8ef3-6415b63b344e None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:18.650247 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fa0713b-a256-438d-8ef3-6415b63b344e None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:18.650548 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fa0713b-a256-438d-8ef3-6415b63b344e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:18.651949 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fa0713b-a256-438d-8ef3-6415b63b344e None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:18.658504 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 371/768] 10.222.0.102 () {56 vars in 1129 bytes} [Wed Aug 24 15:34:18 2022] GET /identity/v3/services?type=block-storage => generated 369 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:18.659945 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4306f2d-0b9b-4207-bca0-508d25d8d118 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:18.660233 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4306f2d-0b9b-4207-bca0-508d25d8d118 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:18.660510 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4306f2d-0b9b-4207-bca0-508d25d8d118 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:18.662066 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 372/769] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:18 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:18.670895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4306f2d-0b9b-4207-bca0-508d25d8d118 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:18.678924 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:18.734219 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:18.760768 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74357d3-43d2-41b3-a3e1-8ce3626704a0 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:18.787537 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4306f2d-0b9b-4207-bca0-508d25d8d118 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:18.800511 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d458fa-317c-4803-8931-76e92e2f40cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:18.801446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d458fa-317c-4803-8931-76e92e2f40cd None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:18.801915 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d458fa-317c-4803-8931-76e92e2f40cd None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:18.802342 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d458fa-317c-4803-8931-76e92e2f40cd None admin] PATH_INFO: `/v3/projects/service` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:18.810148 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d458fa-317c-4803-8931-76e92e2f40cd None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:18.812755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d458fa-317c-4803-8931-76e92e2f40cd None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:18.825576 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-b6d458fa-317c-4803-8931-76e92e2f40cd None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:18.825576 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:18.825576 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:18.825576 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:18.825576 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:18.825576 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:18.825576 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:18.825576 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:18.825576 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:18.825576 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:18.825576 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:18.825576 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:18.825576 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:34:18.825576 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:34:18.825576 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:34:18.826570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:34:18.826570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:18.826570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:18.826570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:18.826570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:18.826570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:18.826570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:18.826570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:18.826570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:18.826570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:18.826570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:18.826570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:18.826570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:18.826570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:18.826570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:18.826570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:18.826570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:18.827475 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:34:18.827475 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:34:18.827475 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:34:18.827475 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:34:18.827475 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:34:18.827475 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:34:18.827475 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:18.827475 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:18.830094 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b7b294-fac7-4375-8326-c9c6a08129db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:18.830922 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b7b294-fac7-4375-8326-c9c6a08129db None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:18.831381 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b7b294-fac7-4375-8326-c9c6a08129db None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:18.831683 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 398/770] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:34:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:18.832013 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b7b294-fac7-4375-8326-c9c6a08129db None admin] PATH_INFO: `/v3/endpoints` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:18.832895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b7b294-fac7-4375-8326-c9c6a08129db None admin] RBAC: Adding query filter params (interface=public, service_id=6db204bfff024d5eb6b68f3781990b82, region_id=RegionOne) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:18.833193 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18f69e5-d16f-46e7-9b45-61b3ac62da35 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:18.833533 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18f69e5-d16f-46e7-9b45-61b3ac62da35 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:18.833719 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b7b294-fac7-4375-8326-c9c6a08129db None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:18.833943 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18f69e5-d16f-46e7-9b45-61b3ac62da35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:18.835435 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b7b294-fac7-4375-8326-c9c6a08129db None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:18.849887 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 373/771] 10.222.0.102 () {56 vars in 1256 bytes} [Wed Aug 24 15:34:18 2022] GET /identity/v3/endpoints?service_id=6db204bfff024d5eb6b68f3781990b82&interface=public®ion_id=RegionOne => generated 195 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:18.852183 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b7b294-fac7-4375-8326-c9c6a08129db None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:18.856734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-c18f69e5-d16f-46e7-9b45-61b3ac62da35 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:18.857446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-c18f69e5-d16f-46e7-9b45-61b3ac62da35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:18.939657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 399/772] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:18 2022] POST /identity/v3/auth/tokens => generated 1003 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:34:18.939657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18f69e5-d16f-46e7-9b45-61b3ac62da35 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:18.954949 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:18.968486 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b7b294-fac7-4375-8326-c9c6a08129db None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:18.988282 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:19.007947 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-118edd5f-fc2d-47a2-a8be-484c09e29e84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:19.008706 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118edd5f-fc2d-47a2-a8be-484c09e29e84 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:19.009168 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118edd5f-fc2d-47a2-a8be-484c09e29e84 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:19.009537 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118edd5f-fc2d-47a2-a8be-484c09e29e84 None admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:19.010395 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-118edd5f-fc2d-47a2-a8be-484c09e29e84 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:19.010772 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-118edd5f-fc2d-47a2-a8be-484c09e29e84 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:19.012319 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-118edd5f-fc2d-47a2-a8be-484c09e29e84 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:19.029479 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 374/773] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:34:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:19.031615 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dce805d-4602-47c9-8dd7-fe4c1afc9244 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:19.032122 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dce805d-4602-47c9-8dd7-fe4c1afc9244 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:19.032571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dce805d-4602-47c9-8dd7-fe4c1afc9244 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:19.033964 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 375/774] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:19 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:19.040648 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dce805d-4602-47c9-8dd7-fe4c1afc9244 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:19.056088 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18f69e5-d16f-46e7-9b45-61b3ac62da35 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:19.093203 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c964b2-a6b5-4ab9-af61-04482339e2f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:19.093742 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c964b2-a6b5-4ab9-af61-04482339e2f6 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:19.094037 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c964b2-a6b5-4ab9-af61-04482339e2f6 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:19.094317 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c964b2-a6b5-4ab9-af61-04482339e2f6 None admin] PATH_INFO: `/v3/services` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:19.095032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c964b2-a6b5-4ab9-af61-04482339e2f6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:19.096536 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c964b2-a6b5-4ab9-af61-04482339e2f6 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:19.107672 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 400/775] 10.222.0.102 () {56 vars in 1092 bytes} [Wed Aug 24 15:34:18 2022] GET /identity/v3/services => generated 776 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:19.107672 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd047b30-83f8-4c17-85ec-1ab143687d39 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:19.107672 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd047b30-83f8-4c17-85ec-1ab143687d39 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:19.107672 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd047b30-83f8-4c17-85ec-1ab143687d39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:19.123889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-fd047b30-83f8-4c17-85ec-1ab143687d39 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:19.125103 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-fd047b30-83f8-4c17-85ec-1ab143687d39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:19.187905 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dce805d-4602-47c9-8dd7-fe4c1afc9244 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:19.251845 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f7ba19-59bf-401e-b133-60f77fc48775 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:19.252612 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f7ba19-59bf-401e-b133-60f77fc48775 None admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:19.252962 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f7ba19-59bf-401e-b133-60f77fc48775 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:19.253332 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f7ba19-59bf-401e-b133-60f77fc48775 None admin] PATH_INFO: `/v3/projects/3e50e6f6c1f44c648337c8d37a8e94a5/users/e0d544379fe44438ad10289b45124374/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:19.255659 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59f7ba19-59bf-401e-b133-60f77fc48775 None admin] RBAC: Authorizing `identity:create_grant(project_id=3e50e6f6c1f44c648337c8d37a8e94a5, user_id=e0d544379fe44438ad10289b45124374, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:19.257109 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 401/776] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:19 2022] POST /identity/v3/auth/tokens => generated 1003 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:34:19.257269 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59f7ba19-59bf-401e-b133-60f77fc48775 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:19.263662 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd047b30-83f8-4c17-85ec-1ab143687d39 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:19.290814 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 376/777] 10.222.0.102 () {58 vars in 1444 bytes} [Wed Aug 24 15:34:19 2022] PUT /identity/v3/projects/3e50e6f6c1f44c648337c8d37a8e94a5/users/e0d544379fe44438ad10289b45124374/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:34:19.309771 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:19.419087 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd047b30-83f8-4c17-85ec-1ab143687d39 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:19.460296 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1096a3-0a04-4736-84c4-e868ebbfae1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:19.461068 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1096a3-0a04-4736-84c4-e868ebbfae1e None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:19.461346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1096a3-0a04-4736-84c4-e868ebbfae1e None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:19.461662 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1096a3-0a04-4736-84c4-e868ebbfae1e None admin] PATH_INFO: `/v3/roles/service` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:19.476504 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c1096a3-0a04-4736-84c4-e868ebbfae1e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:34:19.481568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-0c1096a3-0a04-4736-84c4-e868ebbfae1e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:34:19.481568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:19.481568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:19.481568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:19.481568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:19.481568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:19.481568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:19.481568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:19.481568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:19.481568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:19.481568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:19.481568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:19.481568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:34:19.481568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:34:19.481568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:34:19.481568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:34:19.482588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:34:19.482588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:34:19.482588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:34:19.482588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:34:19.482588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:19.482588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:19.482588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:19.482588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:19.482588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:19.482588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:19.482588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:19.482588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:19.482588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:19.482588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:19.482588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:19.482588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:19.483495 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:19.483495 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:19.483495 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:19.483495 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:19.483495 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:34:19.483495 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:34:19.483495 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:34:19.483495 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:34:19.483495 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:34:19.483495 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:34:19.483495 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:34:19.483495 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:19.483495 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 402/778] 10.222.0.102 () {56 vars in 1107 bytes} [Wed Aug 24 15:34:19 2022] GET /identity/v3/roles/service => generated 85 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:19.489115 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f7ba19-59bf-401e-b133-60f77fc48775 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:19.520411 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052365a9-8c90-4e74-81fb-e868aaddd51c None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:19.520744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052365a9-8c90-4e74-81fb-e868aaddd51c None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:19.521085 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052365a9-8c90-4e74-81fb-e868aaddd51c None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:19.522692 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 403/779] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:19 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:34:19.527509 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad84ff0f-f77a-4009-ba55-b318337863f2 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:19.527821 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad84ff0f-f77a-4009-ba55-b318337863f2 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:19.528098 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad84ff0f-f77a-4009-ba55-b318337863f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:19.535196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:19.554776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-ad84ff0f-f77a-4009-ba55-b318337863f2 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:19.556110 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-ad84ff0f-f77a-4009-ba55-b318337863f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:19.621769 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f7ba19-59bf-401e-b133-60f77fc48775 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:19.663181 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 404/780] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:19 2022] POST /identity/v3/auth/tokens => generated 1003 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 15:34:19.664256 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad84ff0f-f77a-4009-ba55-b318337863f2 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:19.665017 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7f1a79-bae7-4683-bde5-5c2ef3ef1fe2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:19.665869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7f1a79-bae7-4683-bde5-5c2ef3ef1fe2 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:19.666794 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:19.667015 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7f1a79-bae7-4683-bde5-5c2ef3ef1fe2 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:19.668697 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7f1a79-bae7-4683-bde5-5c2ef3ef1fe2 None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:19.670020 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7f1a79-bae7-4683-bde5-5c2ef3ef1fe2 None admin] RBAC: Adding query filter params (name=service) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:19.671382 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7f1a79-bae7-4683-bde5-5c2ef3ef1fe2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:19.673538 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7f1a79-bae7-4683-bde5-5c2ef3ef1fe2 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:19.689051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 377/781] 10.222.0.102 () {56 vars in 1108 bytes} [Wed Aug 24 15:34:19 2022] GET /identity/v3/roles?name=service => generated 328 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:19.695302 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7f1a79-bae7-4683-bde5-5c2ef3ef1fe2 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:19.831706 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad84ff0f-f77a-4009-ba55-b318337863f2 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:19.838228 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7f1a79-bae7-4683-bde5-5c2ef3ef1fe2 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:19.840057 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:19.897729 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7564e04f-3072-48e1-9666-f4e663ec81d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:19.898373 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7564e04f-3072-48e1-9666-f4e663ec81d8 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:19.898651 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7564e04f-3072-48e1-9666-f4e663ec81d8 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:19.898924 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7564e04f-3072-48e1-9666-f4e663ec81d8 None admin] PATH_INFO: `/v3/services/network` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:19.899784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7564e04f-3072-48e1-9666-f4e663ec81d8 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:19.901986 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7564e04f-3072-48e1-9666-f4e663ec81d8 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:19.917656 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa95ae2-56aa-4c45-862a-152c0a3fb3b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:19.918306 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa95ae2-56aa-4c45-862a-152c0a3fb3b1 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:19.918569 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa95ae2-56aa-4c45-862a-152c0a3fb3b1 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:19.918842 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa95ae2-56aa-4c45-862a-152c0a3fb3b1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:19.921115 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-7564e04f-3072-48e1-9666-f4e663ec81d8 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 24 15:34:19.921115 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:19.921115 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 15:34:19.921115 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 15:34:19.921115 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 15:34:19.921115 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 15:34:19.921115 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 15:34:19.921115 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:19.921115 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 24 15:34:19.921115 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:19.921115 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 15:34:19.921115 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:19.921115 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:19.921115 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:19.921115 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:19.921115 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:19.921115 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:19.922675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:19.922675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:19.922675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:19.922675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:19.922675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:19.922675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:19.922675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 15:34:19.922675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 15:34:19.922675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 15:34:19.922675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 15:34:19.922675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:19.922675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:19.922675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:19.922675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:19.922675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:19.922675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:19.924799 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:19.924799 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:19.924799 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:19.924799 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:19.924799 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:19.924799 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:19.924799 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:19.924799 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:19.924799 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:19.924799 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:19.924799 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 15:34:19.924799 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:19.924799 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 24 15:34:19.924799 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:19.924799 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 405/782] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:34:19 2022] GET /identity/v3/services/network => generated 88 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:19.928447 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7564e04f-3072-48e1-9666-f4e663ec81d8 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:19.935903 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa95ae2-56aa-4c45-862a-152c0a3fb3b1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:19.937778 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa95ae2-56aa-4c45-862a-152c0a3fb3b1 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:19.963051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-ffa95ae2-56aa-4c45-862a-152c0a3fb3b1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:19.963051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:19.963051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:19.963051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:19.963051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:19.963051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:19.963051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:19.963051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:19.963051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:19.963051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:19.963051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:19.963051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:19.963051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:19.963051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:19.963051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:19.963051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:19.965885 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:19.965885 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:19.965885 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:19.965885 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:19.965885 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:19.965885 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:19.965885 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:19.965885 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:19.965885 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:19.965885 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:19.965885 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:19.965885 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:19.965885 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:19.965885 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:19.965885 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:19.965885 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:19.965885 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:19.971955 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:19.971955 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:19.971955 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:19.972674 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 378/783] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 272 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:19.974414 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa95ae2-56aa-4c45-862a-152c0a3fb3b1 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:20.059108 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:20.114172 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7564e04f-3072-48e1-9666-f4e663ec81d8 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:20.120171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa95ae2-56aa-4c45-862a-152c0a3fb3b1 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:20.161426 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6197c0eb-41b3-4a15-b76f-e432fc7ebdcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:20.162062 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6197c0eb-41b3-4a15-b76f-e432fc7ebdcf None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:20.162341 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6197c0eb-41b3-4a15-b76f-e432fc7ebdcf None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:20.162615 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6197c0eb-41b3-4a15-b76f-e432fc7ebdcf None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:20.163285 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6197c0eb-41b3-4a15-b76f-e432fc7ebdcf None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:20.163592 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6197c0eb-41b3-4a15-b76f-e432fc7ebdcf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:20.165075 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6197c0eb-41b3-4a15-b76f-e432fc7ebdcf None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:20.184451 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 379/784] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:19 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:20.185327 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6197c0eb-41b3-4a15-b76f-e432fc7ebdcf None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:20.199760 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:20.283474 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9419e1b5-0fa2-45fe-b8c7-4970baba1e64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:20.284892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9419e1b5-0fa2-45fe-b8c7-4970baba1e64 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:20.285599 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9419e1b5-0fa2-45fe-b8c7-4970baba1e64 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:20.286022 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9419e1b5-0fa2-45fe-b8c7-4970baba1e64 None admin] PATH_INFO: `/v3/services` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:20.287056 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9419e1b5-0fa2-45fe-b8c7-4970baba1e64 None admin] RBAC: Adding query filter params (name=network) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:20.287557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9419e1b5-0fa2-45fe-b8c7-4970baba1e64 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:20.289155 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9419e1b5-0fa2-45fe-b8c7-4970baba1e64 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:20.294961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 406/785] 10.222.0.102 () {56 vars in 1117 bytes} [Wed Aug 24 15:34:19 2022] GET /identity/v3/services?name=network => generated 125 bytes in 367 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:20.296070 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9419e1b5-0fa2-45fe-b8c7-4970baba1e64 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:20.304658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6197c0eb-41b3-4a15-b76f-e432fc7ebdcf None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:20.347674 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d665b3-ecb9-46f7-bc37-b1df7c324b26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:20.348464 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d665b3-ecb9-46f7-bc37-b1df7c324b26 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:20.348809 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d665b3-ecb9-46f7-bc37-b1df7c324b26 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:20.349196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d665b3-ecb9-46f7-bc37-b1df7c324b26 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:20.362437 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d665b3-ecb9-46f7-bc37-b1df7c324b26 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:34:20.367616 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-36d665b3-ecb9-46f7-bc37-b1df7c324b26 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:34:20.367616 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:20.367616 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:20.367616 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:20.367616 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:20.367616 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:20.367616 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:20.367616 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:20.367616 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:20.367616 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:20.367616 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:20.367616 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:20.367616 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:34:20.367616 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:34:20.367616 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:34:20.367616 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:34:20.368766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:34:20.368766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:34:20.368766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:34:20.368766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:34:20.368766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:20.368766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:20.368766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:20.368766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:20.368766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:20.368766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:20.368766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:20.368766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:20.368766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:20.368766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:20.368766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:20.368766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:20.369811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:20.369811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:20.369811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:20.369811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:20.369811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:34:20.369811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:34:20.369811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:34:20.369811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:34:20.369811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:34:20.369811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:34:20.369811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:34:20.369811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:20.375216 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 380/786] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:34:20 2022] GET /identity/v3/roles/admin => generated 83 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:20.376204 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d665b3-ecb9-46f7-bc37-b1df7c324b26 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:20.411231 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9419e1b5-0fa2-45fe-b8c7-4970baba1e64 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:20.443256 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:20.450748 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3640ef-d5a2-446f-8aac-f15a3d2c4e9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:20.452280 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3640ef-d5a2-446f-8aac-f15a3d2c4e9b None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:20.452809 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3640ef-d5a2-446f-8aac-f15a3d2c4e9b None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:20.453261 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3640ef-d5a2-446f-8aac-f15a3d2c4e9b None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:20.460432 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:20.463981 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3640ef-d5a2-446f-8aac-f15a3d2c4e9b None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:20.465872 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3640ef-d5a2-446f-8aac-f15a3d2c4e9b None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:20.479248 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-1d3640ef-d5a2-446f-8aac-f15a3d2c4e9b None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 24 15:34:20.479248 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:20.479248 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:20.479248 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:20.479248 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:20.479248 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:20.479248 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:20.479248 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:20.479248 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:20.479248 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:20.479248 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:20.479248 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:20.479248 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:34:20.479248 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:34:20.479248 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:34:20.479248 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:34:20.480273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:20.480273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:20.480273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:34:20.480273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:20.480273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:34:20.480273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:20.480273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:20.480273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:20.480273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:20.480273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:20.480273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:20.480273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:20.480273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:20.480273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:20.480273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:20.480273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:20.480273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:20.480273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:20.481191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:20.481191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:20.481191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:34:20.481191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:34:20.481191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:34:20.481191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:34:20.481191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:34:20.481191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:34:20.481191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 24 15:34:20.481191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:20.488397 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 407/787] 10.222.0.102 () {56 vars in 1122 bytes} [Wed Aug 24 15:34:20 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:20.489721 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3640ef-d5a2-446f-8aac-f15a3d2c4e9b None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:20.495419 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d665b3-ecb9-46f7-bc37-b1df7c324b26 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:20.540140 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6560665-60b8-4a79-b316-c4abf8048e9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:20.540759 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6560665-60b8-4a79-b316-c4abf8048e9c None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:20.541038 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6560665-60b8-4a79-b316-c4abf8048e9c None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:20.541310 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6560665-60b8-4a79-b316-c4abf8048e9c None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:20.541956 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6560665-60b8-4a79-b316-c4abf8048e9c None admin] RBAC: Adding query filter params (type=network) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:20.542265 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6560665-60b8-4a79-b316-c4abf8048e9c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:20.543648 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6560665-60b8-4a79-b316-c4abf8048e9c None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:20.554389 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 381/788] 10.222.0.102 () {56 vars in 1117 bytes} [Wed Aug 24 15:34:20 2022] GET /identity/v3/services?type=network => generated 352 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:20.555331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6560665-60b8-4a79-b316-c4abf8048e9c None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:20.604943 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3640ef-d5a2-446f-8aac-f15a3d2c4e9b None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:20.643177 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aeeb49d-386e-4e9a-a515-5a8d731fc526 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:20.644260 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aeeb49d-386e-4e9a-a515-5a8d731fc526 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:20.644726 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aeeb49d-386e-4e9a-a515-5a8d731fc526 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:20.645279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aeeb49d-386e-4e9a-a515-5a8d731fc526 None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:20.646147 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aeeb49d-386e-4e9a-a515-5a8d731fc526 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:20.646738 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aeeb49d-386e-4e9a-a515-5a8d731fc526 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:20.648489 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aeeb49d-386e-4e9a-a515-5a8d731fc526 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:20.660742 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 408/789] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:34:20 2022] GET /identity/v3/roles?name=admin => generated 341 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:20.662171 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aeeb49d-386e-4e9a-a515-5a8d731fc526 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:20.665711 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6560665-60b8-4a79-b316-c4abf8048e9c None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:20.703835 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052768af-2e10-49a8-bd4e-2e1c4af80c7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:20.704371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052768af-2e10-49a8-bd4e-2e1c4af80c7e None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:20.704651 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052768af-2e10-49a8-bd4e-2e1c4af80c7e None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:20.704922 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052768af-2e10-49a8-bd4e-2e1c4af80c7e None admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:20.705662 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052768af-2e10-49a8-bd4e-2e1c4af80c7e None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:20.705965 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052768af-2e10-49a8-bd4e-2e1c4af80c7e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:20.707395 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052768af-2e10-49a8-bd4e-2e1c4af80c7e None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:20.709173 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:20.730720 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 382/790] 10.222.0.102 () {56 vars in 1154 bytes} [Wed Aug 24 15:34:20 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 386 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:20.731658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052768af-2e10-49a8-bd4e-2e1c4af80c7e None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:20.733391 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:20.778369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aeeb49d-386e-4e9a-a515-5a8d731fc526 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:20.815198 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27dd91a-885e-4449-a075-5e55967bac88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:20.816366 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27dd91a-885e-4449-a075-5e55967bac88 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:20.816878 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27dd91a-885e-4449-a075-5e55967bac88 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:20.817329 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27dd91a-885e-4449-a075-5e55967bac88 None admin] PATH_INFO: `/v3/endpoints` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:20.818225 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d27dd91a-885e-4449-a075-5e55967bac88 None admin] RBAC: Adding query filter params (interface=public, service_id=6d884f652a714b3e8db76b5e5e1aaf1d, region_id=RegionOne) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:20.818713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d27dd91a-885e-4449-a075-5e55967bac88 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:20.820433 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d27dd91a-885e-4449-a075-5e55967bac88 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:20.829672 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 409/791] 10.222.0.102 () {56 vars in 1256 bytes} [Wed Aug 24 15:34:20 2022] GET /identity/v3/endpoints?service_id=6d884f652a714b3e8db76b5e5e1aaf1d&interface=public®ion_id=RegionOne => generated 195 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:20.830933 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27dd91a-885e-4449-a075-5e55967bac88 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:20.840597 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052768af-2e10-49a8-bd4e-2e1c4af80c7e None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:20.877133 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c46380-02ad-48a2-8938-aa1143536bf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:20.877767 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c46380-02ad-48a2-8938-aa1143536bf7 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:20.878050 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c46380-02ad-48a2-8938-aa1143536bf7 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:20.878325 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c46380-02ad-48a2-8938-aa1143536bf7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:20.889808 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7c46380-02ad-48a2-8938-aa1143536bf7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:20.891263 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7c46380-02ad-48a2-8938-aa1143536bf7 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:20.907676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-b7c46380-02ad-48a2-8938-aa1143536bf7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:20.907676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:20.907676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:20.907676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:20.907676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:20.907676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:20.907676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:20.907676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:20.907676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:20.907676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:20.907676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:20.907676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:20.907676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:20.907676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:20.907676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:20.907676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:20.908680 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:20.908680 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:20.908680 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:20.908680 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:20.908680 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:20.908680 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:20.908680 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:20.908680 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:20.908680 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:20.908680 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:20.908680 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:20.908680 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:20.908680 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:20.908680 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:20.908680 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:20.908680 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:20.908680 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:20.909567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:20.909567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:20.909567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:20.909567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 383/792] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:20.909801 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c46380-02ad-48a2-8938-aa1143536bf7 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:20.947509 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27dd91a-885e-4449-a075-5e55967bac88 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:20.966716 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:20.972763 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:20.991959 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b192562-1493-406d-b6b9-cdba024ab000 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:20.993119 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b192562-1493-406d-b6b9-cdba024ab000 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:20.993750 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b192562-1493-406d-b6b9-cdba024ab000 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:20.994171 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b192562-1493-406d-b6b9-cdba024ab000 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:21.005899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b192562-1493-406d-b6b9-cdba024ab000 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:21.007594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b192562-1493-406d-b6b9-cdba024ab000 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:21.024776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-6b192562-1493-406d-b6b9-cdba024ab000 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:21.024776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:21.024776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:21.024776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:21.024776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:21.024776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:21.024776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:21.024776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:21.024776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:21.024776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:21.024776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:21.024776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:21.024776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:21.024776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:21.024776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:21.024776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:21.026015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:21.026015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:21.026015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:21.026015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:21.026015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:21.026015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:21.026015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:21.026015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:21.026015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:21.026015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:21.026015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:21.026015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:21.026015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:21.026015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:21.026015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:21.026015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:21.026015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:21.027092 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:21.027092 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:21.027092 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:21.028937 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 410/793] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:21.030787 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b192562-1493-406d-b6b9-cdba024ab000 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:21.037842 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c46380-02ad-48a2-8938-aa1143536bf7 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:21.081373 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8954df1e-c3cf-4cdb-b375-0bc7a59ee79d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:21.081908 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8954df1e-c3cf-4cdb-b375-0bc7a59ee79d None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:21.082194 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8954df1e-c3cf-4cdb-b375-0bc7a59ee79d None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:21.082469 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8954df1e-c3cf-4cdb-b375-0bc7a59ee79d None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:21.083090 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8954df1e-c3cf-4cdb-b375-0bc7a59ee79d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:21.084596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8954df1e-c3cf-4cdb-b375-0bc7a59ee79d None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:21.090853 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 384/794] 10.222.0.102 () {56 vars in 1092 bytes} [Wed Aug 24 15:34:20 2022] GET /identity/v3/services => generated 776 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:21.092047 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8954df1e-c3cf-4cdb-b375-0bc7a59ee79d None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:21.157309 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b192562-1493-406d-b6b9-cdba024ab000 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:21.197620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8118dff-331f-486b-b74a-7b2b9e65edd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:21.198533 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8118dff-331f-486b-b74a-7b2b9e65edd1 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:21.198960 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8118dff-331f-486b-b74a-7b2b9e65edd1 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:21.199380 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8118dff-331f-486b-b74a-7b2b9e65edd1 None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:21.200237 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8118dff-331f-486b-b74a-7b2b9e65edd1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:21.200696 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8118dff-331f-486b-b74a-7b2b9e65edd1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:21.202607 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8118dff-331f-486b-b74a-7b2b9e65edd1 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:21.216830 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 411/795] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:21 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:21.226359 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8118dff-331f-486b-b74a-7b2b9e65edd1 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:21.296492 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:21.307403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8954df1e-c3cf-4cdb-b375-0bc7a59ee79d None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:21.319140 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:21.364118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed033b1-c703-4572-bd45-9a546008ebb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:21.365290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed033b1-c703-4572-bd45-9a546008ebb8 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:21.365758 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed033b1-c703-4572-bd45-9a546008ebb8 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:21.366444 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed033b1-c703-4572-bd45-9a546008ebb8 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:21.368214 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ed033b1-c703-4572-bd45-9a546008ebb8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:21.368529 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ed033b1-c703-4572-bd45-9a546008ebb8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:21.369918 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ed033b1-c703-4572-bd45-9a546008ebb8 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:21.384713 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 385/796] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:21 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:21.391037 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed033b1-c703-4572-bd45-9a546008ebb8 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:21.408962 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8118dff-331f-486b-b74a-7b2b9e65edd1 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:21.454283 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a369977-6295-4460-a08d-00f03b010b0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:21.454939 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a369977-6295-4460-a08d-00f03b010b0a None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:21.455224 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a369977-6295-4460-a08d-00f03b010b0a None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:21.455493 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a369977-6295-4460-a08d-00f03b010b0a None admin] PATH_INFO: `/v3/users/swift` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:21.469174 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a369977-6295-4460-a08d-00f03b010b0a None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:21.471898 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a369977-6295-4460-a08d-00f03b010b0a None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:21.487973 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-2a369977-6295-4460-a08d-00f03b010b0a None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 24 15:34:21.487973 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:21.487973 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:21.487973 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:21.487973 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:21.487973 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:21.487973 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:21.487973 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:21.487973 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:21.487973 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:21.487973 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:21.487973 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:21.487973 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:34:21.487973 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:34:21.487973 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:34:21.487973 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:34:21.488961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:21.488961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:21.488961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:34:21.488961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:21.488961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:34:21.488961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:21.488961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:21.488961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:21.488961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:21.488961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:21.488961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:21.488961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:21.488961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:21.488961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:21.488961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:21.488961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:21.488961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:21.488961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:21.489925 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:21.489925 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:21.489925 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:34:21.489925 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:34:21.489925 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:34:21.489925 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:34:21.489925 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:34:21.489925 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:34:21.489925 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 24 15:34:21.489925 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:21.489925 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 412/797] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:34:21 2022] GET /identity/v3/users/swift => generated 83 bytes in 264 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:21.491118 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3796a33-c477-4341-a000-48fa832a4611 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:21.491292 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3796a33-c477-4341-a000-48fa832a4611 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:21.491567 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3796a33-c477-4341-a000-48fa832a4611 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:21.493221 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 413/798] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:21 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:21.494742 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3796a33-c477-4341-a000-48fa832a4611 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:21.534616 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed033b1-c703-4572-bd45-9a546008ebb8 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:21.583344 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b1ed6f-88d8-469f-b5d3-15a78ad53d3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:21.584364 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b1ed6f-88d8-469f-b5d3-15a78ad53d3d None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:21.584643 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b1ed6f-88d8-469f-b5d3-15a78ad53d3d None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:21.584915 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b1ed6f-88d8-469f-b5d3-15a78ad53d3d None admin] PATH_INFO: `/v3/projects/service` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:21.595355 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b1ed6f-88d8-469f-b5d3-15a78ad53d3d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:21.596930 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:21.597802 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b1ed6f-88d8-469f-b5d3-15a78ad53d3d None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:21.613785 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-a9b1ed6f-88d8-469f-b5d3-15a78ad53d3d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:21.613785 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:21.613785 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:21.613785 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:21.613785 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:21.613785 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:21.613785 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:21.613785 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:21.613785 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:21.613785 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:21.613785 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:21.613785 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:21.613785 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:34:21.613785 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:34:21.613785 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:34:21.614814 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:34:21.614814 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:21.614814 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:21.614814 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:21.614814 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:21.614814 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:21.614814 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:21.614814 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:21.614814 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:21.614814 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:21.614814 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:21.614814 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:21.614814 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:21.614814 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:21.614814 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:21.614814 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:21.614814 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:21.616526 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:34:21.616526 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:34:21.616526 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:34:21.616526 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:34:21.616526 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:34:21.616526 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:34:21.616526 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:21.616526 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:21.616954 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3796a33-c477-4341-a000-48fa832a4611 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:21.617027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 386/799] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:34:21 2022] GET /identity/v3/projects/service => generated 88 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:34:21.617027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fcda8d-2980-4c13-b528-7fdf39ec49e0 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:21.617027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fcda8d-2980-4c13-b528-7fdf39ec49e0 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:21.617027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fcda8d-2980-4c13-b528-7fdf39ec49e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:21.634837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-c6fcda8d-2980-4c13-b528-7fdf39ec49e0 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:21.636048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-c6fcda8d-2980-4c13-b528-7fdf39ec49e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:21.656900 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b385ff46-9d9a-438d-b114-6fbb1335abc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:21.657522 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b385ff46-9d9a-438d-b114-6fbb1335abc3 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:21.657811 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b385ff46-9d9a-438d-b114-6fbb1335abc3 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:21.658082 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b385ff46-9d9a-438d-b114-6fbb1335abc3 None admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:21.658814 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b385ff46-9d9a-438d-b114-6fbb1335abc3 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:21.659135 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b385ff46-9d9a-438d-b114-6fbb1335abc3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:21.660747 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b385ff46-9d9a-438d-b114-6fbb1335abc3 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:21.662598 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:21.682499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 414/800] 10.222.0.102 () {56 vars in 1140 bytes} [Wed Aug 24 15:34:21 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 372 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:21.683879 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c7439b-4c65-43fd-85ad-a2adbde0c056 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:21.684604 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c7439b-4c65-43fd-85ad-a2adbde0c056 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:21.684886 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c7439b-4c65-43fd-85ad-a2adbde0c056 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:21.687054 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 415/801] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:21 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:21.687858 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c7439b-4c65-43fd-85ad-a2adbde0c056 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:21.713870 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 387/802] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:21 2022] POST /identity/v3/auth/tokens => generated 1003 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:34:21.715378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6fcda8d-2980-4c13-b528-7fdf39ec49e0 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:21.817101 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c7439b-4c65-43fd-85ad-a2adbde0c056 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:21.831069 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6fcda8d-2980-4c13-b528-7fdf39ec49e0 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:21.857843 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec03822-9cd1-4ecb-8495-58eb2650e14c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:21.859235 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec03822-9cd1-4ecb-8495-58eb2650e14c None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:21.859722 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec03822-9cd1-4ecb-8495-58eb2650e14c None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:21.860168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec03822-9cd1-4ecb-8495-58eb2650e14c None admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:21.861065 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec03822-9cd1-4ecb-8495-58eb2650e14c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:21.861643 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec03822-9cd1-4ecb-8495-58eb2650e14c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:21.863429 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec03822-9cd1-4ecb-8495-58eb2650e14c None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:21.867156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7becc0-0da9-447b-88e2-c640b118b76a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:21.867903 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7becc0-0da9-447b-88e2-c640b118b76a None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:21.868254 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7becc0-0da9-447b-88e2-c640b118b76a None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:21.868601 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7becc0-0da9-447b-88e2-c640b118b76a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:21.878602 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 416/803] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:34:21 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:21.880165 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cf57d2-f7cc-4dc4-a2cd-85019b06b0ee None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:21.880617 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cf57d2-f7cc-4dc4-a2cd-85019b06b0ee None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:21.881079 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cf57d2-f7cc-4dc4-a2cd-85019b06b0ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:21.884940 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7becc0-0da9-447b-88e2-c640b118b76a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:21.886464 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7becc0-0da9-447b-88e2-c640b118b76a None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:21.896528 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:21.902549 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-8a7becc0-0da9-447b-88e2-c640b118b76a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:21.902549 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:21.902549 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:21.902549 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:21.902549 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:21.902549 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:21.902549 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:21.902549 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:21.902549 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:21.902549 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:21.902549 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:21.902549 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:21.902549 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:21.902549 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:21.902549 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:21.902549 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:21.903536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:21.903536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:21.903536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:21.903536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:21.903536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:21.903536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:21.903536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:21.903536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:21.903536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:21.903536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:21.903536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:21.903536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:21.903536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:21.903536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:21.903536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:21.903536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:21.903536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:21.904478 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:21.904478 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:21.904478 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:21.904478 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 388/804] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:21.904478 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7becc0-0da9-447b-88e2-c640b118b76a None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:21.907570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-d5cf57d2-f7cc-4dc4-a2cd-85019b06b0ee None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:21.909200 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-d5cf57d2-f7cc-4dc4-a2cd-85019b06b0ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:21.918676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:21.994062 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 417/805] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:21 2022] POST /identity/v3/auth/tokens => generated 1003 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:34:21.995564 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5cf57d2-f7cc-4dc4-a2cd-85019b06b0ee None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:22.014982 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7becc0-0da9-447b-88e2-c640b118b76a None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:22.050935 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63aff8b-24af-44bd-a2f5-675b22a3491c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:22.051883 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63aff8b-24af-44bd-a2f5-675b22a3491c None admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:22.052146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63aff8b-24af-44bd-a2f5-675b22a3491c None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:22.052535 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63aff8b-24af-44bd-a2f5-675b22a3491c None admin] PATH_INFO: `/v3/projects/3e50e6f6c1f44c648337c8d37a8e94a5/users/a6c17620ead14cb49d962990e1b1e0d8/roles/628bab09d1dc48fd9e01023b2f7671f6` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:22.068162 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a63aff8b-24af-44bd-a2f5-675b22a3491c None admin] RBAC: Authorizing `identity:create_grant(project_id=3e50e6f6c1f44c648337c8d37a8e94a5, user_id=a6c17620ead14cb49d962990e1b1e0d8, role_id=628bab09d1dc48fd9e01023b2f7671f6)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:22.069790 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a63aff8b-24af-44bd-a2f5-675b22a3491c None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:22.091802 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 389/806] 10.222.0.102 () {58 vars in 1444 bytes} [Wed Aug 24 15:34:21 2022] PUT /identity/v3/projects/3e50e6f6c1f44c648337c8d37a8e94a5/users/a6c17620ead14cb49d962990e1b1e0d8/roles/628bab09d1dc48fd9e01023b2f7671f6 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:34:22.094285 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63aff8b-24af-44bd-a2f5-675b22a3491c None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:22.116386 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5cf57d2-f7cc-4dc4-a2cd-85019b06b0ee None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:22.158422 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe4ed33-4c64-4752-9a6d-7509159d6ee3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:22.159369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe4ed33-4c64-4752-9a6d-7509159d6ee3 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:22.159870 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe4ed33-4c64-4752-9a6d-7509159d6ee3 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:22.160288 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe4ed33-4c64-4752-9a6d-7509159d6ee3 None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:22.161180 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe4ed33-4c64-4752-9a6d-7509159d6ee3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:22.161639 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe4ed33-4c64-4752-9a6d-7509159d6ee3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:22.163228 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe4ed33-4c64-4752-9a6d-7509159d6ee3 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:22.166479 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:22.180646 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 418/807] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:21 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:22.186183 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe4ed33-4c64-4752-9a6d-7509159d6ee3 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:22.230687 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:22.287523 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63aff8b-24af-44bd-a2f5-675b22a3491c None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:22.311991 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe4ed33-4c64-4752-9a6d-7509159d6ee3 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:22.326108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd6b7d3-91e0-4e69-a8de-2ce733972ac5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:22.327172 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd6b7d3-91e0-4e69-a8de-2ce733972ac5 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:22.327453 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd6b7d3-91e0-4e69-a8de-2ce733972ac5 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:22.327754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd6b7d3-91e0-4e69-a8de-2ce733972ac5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:22.346012 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd6b7d3-91e0-4e69-a8de-2ce733972ac5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:34:22.352267 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-cbd6b7d3-91e0-4e69-a8de-2ce733972ac5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:34:22.352267 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:22.352267 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:22.352267 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:22.352267 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:22.352267 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:22.352267 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:22.352267 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:22.352267 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:22.352267 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:22.352267 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:22.352267 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:22.352267 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:34:22.352267 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:34:22.352267 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:34:22.352267 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:34:22.353415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:34:22.353415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:34:22.353415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:34:22.353415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:34:22.353415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:22.353415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:22.353415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:22.353415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:22.353415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:22.353415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:22.353415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:22.353415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:22.353415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:22.353415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:22.353415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:22.353415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:22.354454 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:22.354454 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:22.354454 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:22.354454 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:22.354454 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:34:22.354454 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:34:22.354454 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:34:22.354454 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:34:22.354454 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:34:22.354454 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:34:22.354454 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:34:22.354454 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:22.354454 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 390/808] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:34:22 2022] GET /identity/v3/roles/admin => generated 83 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:22.355337 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd6b7d3-91e0-4e69-a8de-2ce733972ac5 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:22.363655 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca08b51-f99a-4e5e-a910-424be46350f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:22.363655 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca08b51-f99a-4e5e-a910-424be46350f7 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:22.363655 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca08b51-f99a-4e5e-a910-424be46350f7 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:22.363655 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca08b51-f99a-4e5e-a910-424be46350f7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:22.377859 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ca08b51-f99a-4e5e-a910-424be46350f7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:22.379887 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ca08b51-f99a-4e5e-a910-424be46350f7 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:22.396046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-4ca08b51-f99a-4e5e-a910-424be46350f7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:22.396046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:22.396046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:22.396046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:22.396046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:22.396046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:22.396046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:22.396046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:22.396046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:22.396046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:22.396046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:22.396046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:22.396046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:34:22.396046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:34:22.396046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:34:22.397191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:34:22.397191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:22.397191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:22.397191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:22.397191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:22.397191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:22.397191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:22.397191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:22.397191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:22.397191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:22.397191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:22.397191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:22.397191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:22.397191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:22.397191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:22.397191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:22.397191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:22.398312 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:34:22.398312 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:34:22.398312 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:34:22.398312 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:34:22.398312 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:34:22.398312 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:34:22.398312 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:22.398312 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:22.399721 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 419/809] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:34:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:22.400674 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca08b51-f99a-4e5e-a910-424be46350f7 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:22.474713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:22.490156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd6b7d3-91e0-4e69-a8de-2ce733972ac5 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:22.529491 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4d7a23-aa0c-4cc9-856a-008e5918aad9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:22.530829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4d7a23-aa0c-4cc9-856a-008e5918aad9 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:22.531481 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4d7a23-aa0c-4cc9-856a-008e5918aad9 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:22.531904 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4d7a23-aa0c-4cc9-856a-008e5918aad9 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:22.532695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca4d7a23-aa0c-4cc9-856a-008e5918aad9 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:22.534298 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca4d7a23-aa0c-4cc9-856a-008e5918aad9 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:22.540194 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca08b51-f99a-4e5e-a910-424be46350f7 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:22.543960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-ca4d7a23-aa0c-4cc9-856a-008e5918aad9 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 24 15:34:22.543960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:22.543960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 15:34:22.543960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 15:34:22.543960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 15:34:22.543960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 15:34:22.543960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 15:34:22.543960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:22.543960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 24 15:34:22.543960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:22.543960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 15:34:22.543960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:22.543960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:22.543960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:22.543960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:22.543960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:22.543960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:22.545146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:22.545146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:22.545146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:22.545146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:22.545146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:22.545146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:22.545146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 15:34:22.545146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 15:34:22.545146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 15:34:22.545146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 15:34:22.545146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:22.545146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:22.545146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:22.545146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:22.545146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:22.545146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:22.546775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:22.546775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:22.546775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:22.546775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:22.546775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:22.546775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:22.546775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:22.546775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:22.546775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:22.546775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:22.546775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 15:34:22.546775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:22.546775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 24 15:34:22.546775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:22.553352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 391/810] 10.222.0.102 () {56 vars in 1134 bytes} [Wed Aug 24 15:34:22 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:22.553352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4d7a23-aa0c-4cc9-856a-008e5918aad9 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:22.553352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:22.585283 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f83d3a-4eab-4fc0-bad0-cb7aea691885 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:22.585801 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f83d3a-4eab-4fc0-bad0-cb7aea691885 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:22.586189 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f83d3a-4eab-4fc0-bad0-cb7aea691885 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:22.586555 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f83d3a-4eab-4fc0-bad0-cb7aea691885 None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:22.587323 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4f83d3a-4eab-4fc0-bad0-cb7aea691885 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:22.587775 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4f83d3a-4eab-4fc0-bad0-cb7aea691885 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:22.589256 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4f83d3a-4eab-4fc0-bad0-cb7aea691885 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:22.606494 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 420/811] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:34:22 2022] GET /identity/v3/roles?name=admin => generated 341 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:22.606494 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f83d3a-4eab-4fc0-bad0-cb7aea691885 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:22.681509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4d7a23-aa0c-4cc9-856a-008e5918aad9 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:22.719518 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db41e0c-6172-4beb-b536-e525149c80d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:22.720419 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db41e0c-6172-4beb-b536-e525149c80d9 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:22.720861 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db41e0c-6172-4beb-b536-e525149c80d9 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:22.721273 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db41e0c-6172-4beb-b536-e525149c80d9 None admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:22.722113 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db41e0c-6172-4beb-b536-e525149c80d9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:22.722368 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f83d3a-4eab-4fc0-bad0-cb7aea691885 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:22.722722 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db41e0c-6172-4beb-b536-e525149c80d9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:22.724454 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db41e0c-6172-4beb-b536-e525149c80d9 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:22.743400 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 392/812] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:34:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:22.744035 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:22.745342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db41e0c-6172-4beb-b536-e525149c80d9 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:22.763342 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436729a7-7ff6-4946-b8c4-831b5a35d6f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:22.764046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436729a7-7ff6-4946-b8c4-831b5a35d6f8 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:22.764510 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436729a7-7ff6-4946-b8c4-831b5a35d6f8 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:22.764933 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436729a7-7ff6-4946-b8c4-831b5a35d6f8 None admin] PATH_INFO: `/v3/services` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:22.765739 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-436729a7-7ff6-4946-b8c4-831b5a35d6f8 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:22.766046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-436729a7-7ff6-4946-b8c4-831b5a35d6f8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:22.767619 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-436729a7-7ff6-4946-b8c4-831b5a35d6f8 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:22.778535 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 421/813] 10.222.0.102 () {56 vars in 1129 bytes} [Wed Aug 24 15:34:22 2022] GET /identity/v3/services?name=block-storage => generated 131 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:22.778535 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436729a7-7ff6-4946-b8c4-831b5a35d6f8 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:22.809516 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:22.876608 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db41e0c-6172-4beb-b536-e525149c80d9 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:22.904524 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436729a7-7ff6-4946-b8c4-831b5a35d6f8 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:22.917662 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fab1fa-2922-4b49-bf4c-39dd58e3f247 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:22.918647 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fab1fa-2922-4b49-bf4c-39dd58e3f247 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:22.919069 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fab1fa-2922-4b49-bf4c-39dd58e3f247 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:22.919480 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fab1fa-2922-4b49-bf4c-39dd58e3f247 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:22.934631 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33fab1fa-2922-4b49-bf4c-39dd58e3f247 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:22.937002 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33fab1fa-2922-4b49-bf4c-39dd58e3f247 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:22.946255 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32347bca-a206-46ea-9efd-7e0f3d173e0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:22.947187 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32347bca-a206-46ea-9efd-7e0f3d173e0f None admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:22.947610 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32347bca-a206-46ea-9efd-7e0f3d173e0f None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:22.948085 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32347bca-a206-46ea-9efd-7e0f3d173e0f None admin] PATH_INFO: `/v3/projects/3e50e6f6c1f44c648337c8d37a8e94a5/users/6a070d4fd8c4466c857bd03965c11349/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:22.950141 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32347bca-a206-46ea-9efd-7e0f3d173e0f None admin] RBAC: Authorizing `identity:create_grant(project_id=3e50e6f6c1f44c648337c8d37a8e94a5, user_id=6a070d4fd8c4466c857bd03965c11349, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:22.952331 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32347bca-a206-46ea-9efd-7e0f3d173e0f None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:22.955048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-33fab1fa-2922-4b49-bf4c-39dd58e3f247 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:22.955048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:22.955048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:22.955048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:22.955048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:22.955048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:22.955048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:22.955048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:22.955048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:22.955048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:22.955048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:22.955048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:22.955048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:22.955048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:22.955048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:22.955048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:22.956241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:22.956241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:22.956241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:22.956241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:22.956241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:22.956241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:22.956241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:22.956241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:22.956241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:22.956241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:22.956241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:22.956241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:22.956241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:22.956241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:22.956241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:22.956241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:22.956241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:22.961878 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:22.961878 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:22.961878 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:22.961878 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 393/814] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:22.961878 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fab1fa-2922-4b49-bf4c-39dd58e3f247 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:22.981222 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 422/815] 10.222.0.102 () {58 vars in 1444 bytes} [Wed Aug 24 15:34:22 2022] PUT /identity/v3/projects/3e50e6f6c1f44c648337c8d37a8e94a5/users/6a070d4fd8c4466c857bd03965c11349/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:34:22.983154 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32347bca-a206-46ea-9efd-7e0f3d173e0f None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:23.037195 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:23.106667 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32347bca-a206-46ea-9efd-7e0f3d173e0f None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:23.147228 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da6ea89-2b72-421a-b59c-3f75b7b3babe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:23.148227 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da6ea89-2b72-421a-b59c-3f75b7b3babe None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:23.148710 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da6ea89-2b72-421a-b59c-3f75b7b3babe None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:23.149225 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da6ea89-2b72-421a-b59c-3f75b7b3babe None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:23.150191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da6ea89-2b72-421a-b59c-3f75b7b3babe None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:23.150699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da6ea89-2b72-421a-b59c-3f75b7b3babe None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:23.152367 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da6ea89-2b72-421a-b59c-3f75b7b3babe None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:23.165720 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:23.175888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 423/816] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:22 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:23.177390 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fab1fa-2922-4b49-bf4c-39dd58e3f247 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:23.177937 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da6ea89-2b72-421a-b59c-3f75b7b3babe None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:23.244302 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf2600f-fe8b-425a-8ec6-8549336d2d93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:23.245134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf2600f-fe8b-425a-8ec6-8549336d2d93 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:23.245338 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf2600f-fe8b-425a-8ec6-8549336d2d93 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:23.245569 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf2600f-fe8b-425a-8ec6-8549336d2d93 None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:23.246229 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf2600f-fe8b-425a-8ec6-8549336d2d93 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:23.246534 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf2600f-fe8b-425a-8ec6-8549336d2d93 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:23.247936 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf2600f-fe8b-425a-8ec6-8549336d2d93 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:23.271672 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 394/817] 10.222.0.102 () {56 vars in 1129 bytes} [Wed Aug 24 15:34:22 2022] GET /identity/v3/services?type=block-storage => generated 369 bytes in 309 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:23.292798 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf2600f-fe8b-425a-8ec6-8549336d2d93 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:23.311914 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da6ea89-2b72-421a-b59c-3f75b7b3babe None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:23.314548 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:23.351543 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e656c443-a294-49c6-a9fc-1923f256adec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:23.352638 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e656c443-a294-49c6-a9fc-1923f256adec None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:23.353204 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e656c443-a294-49c6-a9fc-1923f256adec None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:23.353677 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e656c443-a294-49c6-a9fc-1923f256adec None admin] PATH_INFO: `/v3/users/nova` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:23.363100 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e656c443-a294-49c6-a9fc-1923f256adec None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:23.365200 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e656c443-a294-49c6-a9fc-1923f256adec None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:23.378889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-e656c443-a294-49c6-a9fc-1923f256adec None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 24 15:34:23.378889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:23.378889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:23.378889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:23.378889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:23.378889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:23.378889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:23.378889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:23.378889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:23.378889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:23.378889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:23.378889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:23.378889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:34:23.378889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:34:23.378889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:34:23.378889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:34:23.379908 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:23.379908 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:23.379908 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:34:23.379908 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:23.379908 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:34:23.379908 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:23.379908 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:23.379908 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:23.379908 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:23.379908 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:23.379908 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:23.379908 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:23.379908 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:23.379908 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:23.379908 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:23.379908 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:23.379908 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:23.379908 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:23.380848 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:23.380848 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:23.380848 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:34:23.380848 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:34:23.380848 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:34:23.380848 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:34:23.380848 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:34:23.380848 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:34:23.380848 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 24 15:34:23.380848 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:23.384568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 424/818] 10.222.0.102 () {56 vars in 1098 bytes} [Wed Aug 24 15:34:23 2022] GET /identity/v3/users/nova => generated 82 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:34:23.390007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e656c443-a294-49c6-a9fc-1923f256adec None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:23.415068 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf2600f-fe8b-425a-8ec6-8549336d2d93 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:23.451909 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3f9484-17ea-42f3-b6af-a1d6a2e73c98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:23.452519 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3f9484-17ea-42f3-b6af-a1d6a2e73c98 None admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:23.452794 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3f9484-17ea-42f3-b6af-a1d6a2e73c98 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:23.453060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3f9484-17ea-42f3-b6af-a1d6a2e73c98 None admin] PATH_INFO: `/v3/endpoints` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:23.453727 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b3f9484-17ea-42f3-b6af-a1d6a2e73c98 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:23.455139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b3f9484-17ea-42f3-b6af-a1d6a2e73c98 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:23.482879 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 395/819] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:34:23 2022] POST /identity/v3/endpoints => generated 348 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:34:23.482879 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a080f0d4-8355-4c0a-bb87-42516bc76560 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:23.482879 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a080f0d4-8355-4c0a-bb87-42516bc76560 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:23.482879 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a080f0d4-8355-4c0a-bb87-42516bc76560 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:23.482879 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 396/820] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:23 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:23.489313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9c13d3-cf53-48f4-9151-6700e5dade6a None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:23.489639 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9c13d3-cf53-48f4-9151-6700e5dade6a None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:23.489919 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9c13d3-cf53-48f4-9151-6700e5dade6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:23.524910 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-ce9c13d3-cf53-48f4-9151-6700e5dade6a None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:23.526138 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-ce9c13d3-cf53-48f4-9151-6700e5dade6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:23.551110 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:23.580967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e656c443-a294-49c6-a9fc-1923f256adec None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:23.671227 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b40126-2b9e-473c-8da5-30d5f1bf5786 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:23.671926 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b40126-2b9e-473c-8da5-30d5f1bf5786 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:23.672307 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b40126-2b9e-473c-8da5-30d5f1bf5786 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:23.672584 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b40126-2b9e-473c-8da5-30d5f1bf5786 None admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:23.673326 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b40126-2b9e-473c-8da5-30d5f1bf5786 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:23.673633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b40126-2b9e-473c-8da5-30d5f1bf5786 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:23.675271 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b40126-2b9e-473c-8da5-30d5f1bf5786 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:23.683078 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 397/821] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:23 2022] POST /identity/v3/auth/tokens => generated 1003 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:34:23.696943 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 425/822] 10.222.0.102 () {56 vars in 1138 bytes} [Wed Aug 24 15:34:23 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 370 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:23.703685 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9c13d3-cf53-48f4-9151-6700e5dade6a None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:23.838230 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9c13d3-cf53-48f4-9151-6700e5dade6a None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:23.880020 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea83f71-ca4a-4f96-9ad3-7a4263f0575d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:23.880638 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea83f71-ca4a-4f96-9ad3-7a4263f0575d None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:23.880924 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea83f71-ca4a-4f96-9ad3-7a4263f0575d None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:23.881195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea83f71-ca4a-4f96-9ad3-7a4263f0575d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:23.901763 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:23.994296 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea83f71-ca4a-4f96-9ad3-7a4263f0575d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:23.995786 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea83f71-ca4a-4f96-9ad3-7a4263f0575d None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:24.039948 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b40126-2b9e-473c-8da5-30d5f1bf5786 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:24.081867 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:24.111329 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 398/823] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:34:23 2022] GET /identity/v3/auth/tokens => generated 1003 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 15:34:24.117221 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea83f71-ca4a-4f96-9ad3-7a4263f0575d None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:24.211995 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:24.251754 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b40126-2b9e-473c-8da5-30d5f1bf5786 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:24.300370 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea83f71-ca4a-4f96-9ad3-7a4263f0575d None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:24.302221 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95952ec-fdb0-449c-8509-38b49b58c1e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:24.302870 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95952ec-fdb0-449c-8509-38b49b58c1e8 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:24.303149 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95952ec-fdb0-449c-8509-38b49b58c1e8 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:24.303419 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95952ec-fdb0-449c-8509-38b49b58c1e8 None admin] PATH_INFO: `/v3/services/network` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:24.304153 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95952ec-fdb0-449c-8509-38b49b58c1e8 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:24.305528 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95952ec-fdb0-449c-8509-38b49b58c1e8 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:24.317609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-c95952ec-fdb0-449c-8509-38b49b58c1e8 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 24 15:34:24.317609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:24.317609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 15:34:24.317609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 15:34:24.317609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 15:34:24.317609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 15:34:24.317609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 15:34:24.317609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:24.317609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 24 15:34:24.317609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:24.317609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 15:34:24.317609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:24.317609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:24.317609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:24.317609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:24.317609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:24.317609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:24.318717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:24.318717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:24.318717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:24.318717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:24.318717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:24.318717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:24.318717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 15:34:24.318717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 15:34:24.318717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 15:34:24.318717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 15:34:24.318717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:24.318717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:24.318717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:24.318717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:24.318717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:24.318717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:24.319616 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:24.319616 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:24.319616 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:24.319616 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:24.319616 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:24.319616 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:24.319616 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:24.319616 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:24.319616 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:24.319616 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:24.319616 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 15:34:24.319616 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:24.319616 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 24 15:34:24.319616 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:24.319616 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 426/824] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:34:24 2022] GET /identity/v3/services/network => generated 88 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:34:24.327666 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc91ce7-afaa-4af7-8797-9d54a0b3a067 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:24.327666 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc91ce7-afaa-4af7-8797-9d54a0b3a067 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:24.327666 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc91ce7-afaa-4af7-8797-9d54a0b3a067 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:24.328275 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 427/825] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:24 2022] GET /identity => generated 271 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:24.329147 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc91ce7-afaa-4af7-8797-9d54a0b3a067 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:24.344309 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb113c8-22ce-4815-badb-4bb7fb12fe19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:24.344948 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb113c8-22ce-4815-badb-4bb7fb12fe19 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:24.345234 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb113c8-22ce-4815-badb-4bb7fb12fe19 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:24.345508 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb113c8-22ce-4815-badb-4bb7fb12fe19 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:24.356740 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cb113c8-22ce-4815-badb-4bb7fb12fe19 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:24.359257 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cb113c8-22ce-4815-badb-4bb7fb12fe19 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:24.375477 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-4cb113c8-22ce-4815-badb-4bb7fb12fe19 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:24.375477 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:24.375477 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:24.375477 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:24.375477 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:24.375477 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:24.375477 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:24.375477 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:24.375477 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:24.375477 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:24.375477 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:24.375477 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:24.375477 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:24.375477 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:24.375477 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:24.375477 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:24.376546 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:24.376546 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:24.376546 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:24.376546 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:24.376546 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:24.376546 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:24.376546 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:24.376546 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:24.376546 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:24.376546 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:24.376546 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:24.376546 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:24.376546 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:24.376546 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:24.376546 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:24.376546 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:24.376546 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:24.377442 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:24.377442 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:24.377442 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:24.377442 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 399/826] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:24.378256 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a3465b-75c7-4e16-8ae8-6838eccfe169 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:24.378548 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a3465b-75c7-4e16-8ae8-6838eccfe169 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:24.378987 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a3465b-75c7-4e16-8ae8-6838eccfe169 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:24.398764 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-69a3465b-75c7-4e16-8ae8-6838eccfe169 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:24.400109 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-69a3465b-75c7-4e16-8ae8-6838eccfe169 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:24.446587 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc91ce7-afaa-4af7-8797-9d54a0b3a067 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:24.452286 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:24.481392 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 400/827] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:24 2022] POST /identity/v3/auth/tokens => generated 1003 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:34:24.482365 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a3465b-75c7-4e16-8ae8-6838eccfe169 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:24.508011 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c45399-90e4-4748-ad94-dc3b6967d1c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:24.508574 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c45399-90e4-4748-ad94-dc3b6967d1c1 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:24.510104 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c45399-90e4-4748-ad94-dc3b6967d1c1 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:24.510383 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c45399-90e4-4748-ad94-dc3b6967d1c1 None admin] PATH_INFO: `/v3/services` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:24.511040 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7c45399-90e4-4748-ad94-dc3b6967d1c1 None admin] RBAC: Adding query filter params (name=network) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:24.512889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7c45399-90e4-4748-ad94-dc3b6967d1c1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:24.518673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7c45399-90e4-4748-ad94-dc3b6967d1c1 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:24.528017 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 428/828] 10.222.0.102 () {56 vars in 1117 bytes} [Wed Aug 24 15:34:24 2022] GET /identity/v3/services?name=network => generated 125 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:24.532660 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c45399-90e4-4748-ad94-dc3b6967d1c1 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:24.536616 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:24.662522 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a3465b-75c7-4e16-8ae8-6838eccfe169 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:24.699460 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c45399-90e4-4748-ad94-dc3b6967d1c1 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:24.744556 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcc28f3-8ecd-45e5-89a3-108bd62c8101 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:24.745717 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcc28f3-8ecd-45e5-89a3-108bd62c8101 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:24.747288 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcc28f3-8ecd-45e5-89a3-108bd62c8101 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:24.748035 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcc28f3-8ecd-45e5-89a3-108bd62c8101 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:24.750962 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fcc28f3-8ecd-45e5-89a3-108bd62c8101 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:24.751997 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fcc28f3-8ecd-45e5-89a3-108bd62c8101 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:24.754158 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fcc28f3-8ecd-45e5-89a3-108bd62c8101 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:24.769259 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78abedbf-a0d4-4107-9e58-fe2ad8448bd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:24.770270 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78abedbf-a0d4-4107-9e58-fe2ad8448bd7 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:24.770808 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78abedbf-a0d4-4107-9e58-fe2ad8448bd7 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:24.771329 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78abedbf-a0d4-4107-9e58-fe2ad8448bd7 None admin] PATH_INFO: `/v3/services` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:24.772207 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78abedbf-a0d4-4107-9e58-fe2ad8448bd7 None admin] RBAC: Adding query filter params (type=network) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:24.772745 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78abedbf-a0d4-4107-9e58-fe2ad8448bd7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:24.774387 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78abedbf-a0d4-4107-9e58-fe2ad8448bd7 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:24.785208 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 401/829] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:24 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:24.787921 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 429/830] 10.222.0.102 () {56 vars in 1117 bytes} [Wed Aug 24 15:34:24 2022] GET /identity/v3/services?type=network => generated 352 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:24.790084 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcc28f3-8ecd-45e5-89a3-108bd62c8101 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:24.793729 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78abedbf-a0d4-4107-9e58-fe2ad8448bd7 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:24.816801 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:24.929196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:24.946397 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78abedbf-a0d4-4107-9e58-fe2ad8448bd7 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:24.947146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcc28f3-8ecd-45e5-89a3-108bd62c8101 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:24.984659 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e53fffe-c4c6-4f79-ae74-d20f3885bc92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:24.985582 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e53fffe-c4c6-4f79-ae74-d20f3885bc92 None admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:24.986019 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e53fffe-c4c6-4f79-ae74-d20f3885bc92 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:24.986451 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e53fffe-c4c6-4f79-ae74-d20f3885bc92 None admin] PATH_INFO: `/v3/endpoints` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:24.986635 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-103613d3-48dd-484c-abdc-66a3f0cc5b87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:24.987379 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103613d3-48dd-484c-abdc-66a3f0cc5b87 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:24.987613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e53fffe-c4c6-4f79-ae74-d20f3885bc92 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:24.987879 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103613d3-48dd-484c-abdc-66a3f0cc5b87 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:24.988237 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103613d3-48dd-484c-abdc-66a3f0cc5b87 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:24.989486 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e53fffe-c4c6-4f79-ae74-d20f3885bc92 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:24.996663 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-103613d3-48dd-484c-abdc-66a3f0cc5b87 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:24.998573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-103613d3-48dd-484c-abdc-66a3f0cc5b87 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:25.015564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-103613d3-48dd-484c-abdc-66a3f0cc5b87 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:25.015564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:25.015564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:25.015564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:25.015564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:25.015564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:25.015564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:25.015564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:25.015564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:25.015564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:25.015564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:25.015564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:25.015564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:34:25.015564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:34:25.015564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:34:25.016822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:34:25.016822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:25.016822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:25.016822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:25.016822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:25.016822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:25.016822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:25.016822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:25.016822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:25.016822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:25.016822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:25.016822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:25.016822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:25.016822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:25.016822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:25.016822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:25.016822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:25.017732 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:34:25.017732 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:34:25.017732 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:34:25.017732 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:34:25.017732 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:34:25.017732 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:34:25.017732 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:25.017732 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:25.017732 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 402/831] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:34:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:34:25.018392 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-103613d3-48dd-484c-abdc-66a3f0cc5b87 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:25.020318 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 430/832] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:34:24 2022] POST /identity/v3/endpoints => generated 339 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:34:25.023444 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e53fffe-c4c6-4f79-ae74-d20f3885bc92 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:25.101216 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:25.155304 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e53fffe-c4c6-4f79-ae74-d20f3885bc92 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:25.197296 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714e9509-305e-47fc-bddc-517d87c0cbd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:25.198229 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714e9509-305e-47fc-bddc-517d87c0cbd9 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:25.198652 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714e9509-305e-47fc-bddc-517d87c0cbd9 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:25.199083 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714e9509-305e-47fc-bddc-517d87c0cbd9 None admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:25.199972 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714e9509-305e-47fc-bddc-517d87c0cbd9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:25.200465 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714e9509-305e-47fc-bddc-517d87c0cbd9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:25.202646 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714e9509-305e-47fc-bddc-517d87c0cbd9 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:25.217154 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 431/833] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:34:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:25.225905 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-103613d3-48dd-484c-abdc-66a3f0cc5b87 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:25.226456 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714e9509-305e-47fc-bddc-517d87c0cbd9 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:25.296157 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd242b13-761c-44db-8cb7-69b74f1ed652 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:25.297475 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd242b13-761c-44db-8cb7-69b74f1ed652 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:25.298009 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd242b13-761c-44db-8cb7-69b74f1ed652 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:25.300461 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd242b13-761c-44db-8cb7-69b74f1ed652 None admin] PATH_INFO: `/v3/roles/service` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:25.322088 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd242b13-761c-44db-8cb7-69b74f1ed652 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:34:25.330698 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-cd242b13-761c-44db-8cb7-69b74f1ed652 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:34:25.330698 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:25.330698 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:25.330698 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:25.330698 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:25.330698 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:25.330698 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:25.330698 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:25.330698 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:25.330698 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:25.330698 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:25.330698 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:25.330698 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:34:25.330698 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:34:25.330698 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:34:25.330698 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:34:25.331750 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:34:25.331750 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:34:25.331750 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:34:25.331750 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:34:25.331750 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:25.331750 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:25.331750 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:25.331750 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:25.331750 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:25.331750 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:25.331750 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:25.331750 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:25.331750 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:25.331750 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:25.331750 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:25.331750 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:25.332767 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:25.332767 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:25.332767 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:25.332767 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:25.332767 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:34:25.332767 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:34:25.332767 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:34:25.332767 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:34:25.332767 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:34:25.332767 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:34:25.332767 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:34:25.332767 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:25.335905 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 403/834] 10.222.0.102 () {56 vars in 1107 bytes} [Wed Aug 24 15:34:25 2022] GET /identity/v3/roles/service => generated 85 bytes in 318 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:25.346783 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd242b13-761c-44db-8cb7-69b74f1ed652 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:25.360309 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:25.418495 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714e9509-305e-47fc-bddc-517d87c0cbd9 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:25.450666 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:25.471940 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e73ee13-2c7f-4f50-a506-2fc11f456da6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:25.472818 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e73ee13-2c7f-4f50-a506-2fc11f456da6 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:25.473108 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e73ee13-2c7f-4f50-a506-2fc11f456da6 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:25.473804 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e73ee13-2c7f-4f50-a506-2fc11f456da6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:25.474703 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e73ee13-2c7f-4f50-a506-2fc11f456da6 None admin] RBAC: Adding query filter params (role.id=4d8fbbaea81a4145b89af12350bb067c, scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5, user.id=e0d544379fe44438ad10289b45124374) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:25.475027 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e73ee13-2c7f-4f50-a506-2fc11f456da6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:25.476916 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e73ee13-2c7f-4f50-a506-2fc11f456da6 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:25.480216 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd242b13-761c-44db-8cb7-69b74f1ed652 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:25.490595 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 432/835] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:34:25 2022] GET /identity/v3/role_assignments?user.id=e0d544379fe44438ad10289b45124374&scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5&role.id=4d8fbbaea81a4145b89af12350bb067c => generated 610 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:25.534924 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0174b1e4-185f-4872-8500-014ef8fc974f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:25.535556 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0174b1e4-185f-4872-8500-014ef8fc974f None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:25.535983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0174b1e4-185f-4872-8500-014ef8fc974f None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:25.536366 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0174b1e4-185f-4872-8500-014ef8fc974f None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:25.537063 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0174b1e4-185f-4872-8500-014ef8fc974f None admin] RBAC: Adding query filter params (name=service) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:25.537372 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0174b1e4-185f-4872-8500-014ef8fc974f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:25.538793 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0174b1e4-185f-4872-8500-014ef8fc974f None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:25.551996 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 404/836] 10.222.0.102 () {56 vars in 1108 bytes} [Wed Aug 24 15:34:25 2022] GET /identity/v3/roles?name=service => generated 328 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:25.557392 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e73ee13-2c7f-4f50-a506-2fc11f456da6 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:25.595381 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d246015d-ca99-4e9f-a721-0f952a921c3d None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:25.595987 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d246015d-ca99-4e9f-a721-0f952a921c3d None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:25.596462 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d246015d-ca99-4e9f-a721-0f952a921c3d None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:25.597986 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 405/837] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:25 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:34:25.604493 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fcf3ac4-ccaa-446d-982a-9250bba5359c None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:25.604925 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fcf3ac4-ccaa-446d-982a-9250bba5359c None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:25.605320 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fcf3ac4-ccaa-446d-982a-9250bba5359c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:25.628918 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-4fcf3ac4-ccaa-446d-982a-9250bba5359c None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:25.630602 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-4fcf3ac4-ccaa-446d-982a-9250bba5359c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:25.699744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e73ee13-2c7f-4f50-a506-2fc11f456da6 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:25.716181 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:25.724094 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 406/838] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:25 2022] POST /identity/v3/auth/tokens => generated 1163 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:34:25.742791 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518c76c8-46db-40e2-b7cd-d11119fa1326 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:25.743565 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518c76c8-46db-40e2-b7cd-d11119fa1326 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:25.743983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518c76c8-46db-40e2-b7cd-d11119fa1326 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:25.744350 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518c76c8-46db-40e2-b7cd-d11119fa1326 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:25.757154 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-518c76c8-46db-40e2-b7cd-d11119fa1326 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:25.759197 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-518c76c8-46db-40e2-b7cd-d11119fa1326 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:25.786977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-518c76c8-46db-40e2-b7cd-d11119fa1326 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:25.786977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:25.786977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:25.786977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:25.786977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:25.786977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:25.786977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:25.786977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:25.786977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:25.786977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:25.786977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:25.786977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:25.786977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:25.786977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:25.786977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:25.786977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:25.788090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:25.788090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:25.788090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:25.788090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:25.788090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:25.788090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:25.788090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:25.788090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:25.788090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:25.788090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:25.788090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:25.788090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:25.788090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:25.788090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:25.788090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:25.788090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:25.788090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:25.788982 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:25.788982 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:25.788982 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:25.788982 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 433/839] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:25.795265 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fcf3ac4-ccaa-446d-982a-9250bba5359c None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:25.903690 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafe4d54-33d0-4d9d-8704-b14092668658 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:25.903963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafe4d54-33d0-4d9d-8704-b14092668658 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:25.904241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafe4d54-33d0-4d9d-8704-b14092668658 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:25.906476 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 434/840] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:25 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:34:25.906828 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fcf3ac4-ccaa-446d-982a-9250bba5359c None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:25.912950 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486d8649-8fff-4283-b5f7-70f2628f5c1e None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:25.913509 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486d8649-8fff-4283-b5f7-70f2628f5c1e None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:25.913999 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486d8649-8fff-4283-b5f7-70f2628f5c1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:25.936076 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-486d8649-8fff-4283-b5f7-70f2628f5c1e None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:25.939044 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-486d8649-8fff-4283-b5f7-70f2628f5c1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:25.948902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f396962a-c759-406a-aa50-b1b687dfdd57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:25.949512 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f396962a-c759-406a-aa50-b1b687dfdd57 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:25.949789 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f396962a-c759-406a-aa50-b1b687dfdd57 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:25.950056 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f396962a-c759-406a-aa50-b1b687dfdd57 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:25.950820 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f396962a-c759-406a-aa50-b1b687dfdd57 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:25.951122 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f396962a-c759-406a-aa50-b1b687dfdd57 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:25.953524 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f396962a-c759-406a-aa50-b1b687dfdd57 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:25.971382 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 407/841] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:25 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:25.972844 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f396962a-c759-406a-aa50-b1b687dfdd57 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:25.981053 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:26.030751 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 435/842] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:25 2022] POST /identity/v3/auth/tokens => generated 1163 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 15:34:26.031815 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486d8649-8fff-4283-b5f7-70f2628f5c1e None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:26.044610 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:26.090512 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f396962a-c759-406a-aa50-b1b687dfdd57 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:26.127999 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc73322f-fe57-403c-b235-40485b5c778e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:26.128797 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc73322f-fe57-403c-b235-40485b5c778e None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:26.129173 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc73322f-fe57-403c-b235-40485b5c778e None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:26.129755 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc73322f-fe57-403c-b235-40485b5c778e None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:26.138317 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc73322f-fe57-403c-b235-40485b5c778e None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:26.141441 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc73322f-fe57-403c-b235-40485b5c778e None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:26.150924 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486d8649-8fff-4283-b5f7-70f2628f5c1e None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:26.154431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-dc73322f-fe57-403c-b235-40485b5c778e None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 24 15:34:26.154431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:26.154431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:26.154431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:26.154431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:26.154431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:26.154431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:26.154431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:26.154431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:26.154431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:26.154431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:26.154431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:26.154431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:34:26.154431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:34:26.154431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:34:26.154431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:34:26.156142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:26.156142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:26.156142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:34:26.156142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:26.156142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:34:26.156142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:26.156142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:26.156142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:26.156142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:26.156142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:26.156142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:26.156142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:26.156142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:26.156142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:26.156142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:26.156142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:26.156142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:26.156142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:26.157369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:26.157369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:26.157369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:34:26.157369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:34:26.157369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:34:26.157369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:34:26.157369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:34:26.157369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:34:26.157369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 24 15:34:26.157369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:26.162257 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 408/843] 10.222.0.102 () {56 vars in 1122 bytes} [Wed Aug 24 15:34:25 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:34:26.163240 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc73322f-fe57-403c-b235-40485b5c778e None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:26.194565 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6157980-9dfe-4b37-aed2-4a2a9508a551 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:26.195662 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6157980-9dfe-4b37-aed2-4a2a9508a551 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:26.196172 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6157980-9dfe-4b37-aed2-4a2a9508a551 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:26.196644 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6157980-9dfe-4b37-aed2-4a2a9508a551 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:26.209127 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6157980-9dfe-4b37-aed2-4a2a9508a551 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:34:26.214620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-c6157980-9dfe-4b37-aed2-4a2a9508a551 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:34:26.214620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:26.214620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:26.214620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:26.214620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:26.214620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:26.214620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:26.214620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:26.214620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:26.214620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:26.214620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:26.214620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:26.214620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:34:26.214620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:34:26.214620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:34:26.214620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:34:26.215530 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:34:26.215530 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:34:26.215530 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:34:26.215530 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:34:26.215530 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:26.215530 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:26.215530 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:26.215530 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:26.215530 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:26.215530 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:26.215530 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:26.215530 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:26.215530 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:26.215530 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:26.215530 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:26.215530 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:26.216390 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:26.216390 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:26.216390 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:26.216390 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:26.216390 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:34:26.216390 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:34:26.216390 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:34:26.216390 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:34:26.216390 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:34:26.216390 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:34:26.216390 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:34:26.216390 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:26.219999 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 436/844] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:34:26 2022] GET /identity/v3/roles/admin => generated 83 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:26.231663 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6157980-9dfe-4b37-aed2-4a2a9508a551 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:26.266206 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:26.277393 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc73322f-fe57-403c-b235-40485b5c778e None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:26.304322 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:26.321213 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0fd67a-5436-4a2a-9209-d6e9ac6c3349 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:26.321915 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0fd67a-5436-4a2a-9209-d6e9ac6c3349 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:26.322202 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0fd67a-5436-4a2a-9209-d6e9ac6c3349 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:26.322482 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0fd67a-5436-4a2a-9209-d6e9ac6c3349 None admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:26.323331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca0fd67a-5436-4a2a-9209-d6e9ac6c3349 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:26.323656 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca0fd67a-5436-4a2a-9209-d6e9ac6c3349 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:26.325529 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca0fd67a-5436-4a2a-9209-d6e9ac6c3349 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:26.344645 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 409/845] 10.222.0.102 () {56 vars in 1154 bytes} [Wed Aug 24 15:34:26 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 386 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:26.345577 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0fd67a-5436-4a2a-9209-d6e9ac6c3349 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:26.363254 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6157980-9dfe-4b37-aed2-4a2a9508a551 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:26.398359 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8cd091-aeb8-4ed4-af76-93a0365e2dc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:26.398975 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8cd091-aeb8-4ed4-af76-93a0365e2dc3 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:26.399257 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8cd091-aeb8-4ed4-af76-93a0365e2dc3 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:26.399536 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8cd091-aeb8-4ed4-af76-93a0365e2dc3 None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:26.400306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8cd091-aeb8-4ed4-af76-93a0365e2dc3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:26.400538 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8cd091-aeb8-4ed4-af76-93a0365e2dc3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:26.401903 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8cd091-aeb8-4ed4-af76-93a0365e2dc3 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:26.411831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 437/846] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:34:26 2022] GET /identity/v3/roles?name=admin => generated 341 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:26.412725 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8cd091-aeb8-4ed4-af76-93a0365e2dc3 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:26.464155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0fd67a-5436-4a2a-9209-d6e9ac6c3349 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:26.505951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4502be-0e92-4407-a801-44e88406e725 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:26.506608 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4502be-0e92-4407-a801-44e88406e725 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:26.506975 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4502be-0e92-4407-a801-44e88406e725 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:26.507163 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4502be-0e92-4407-a801-44e88406e725 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:26.507811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc4502be-0e92-4407-a801-44e88406e725 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:26.509146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc4502be-0e92-4407-a801-44e88406e725 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:26.512585 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:26.519032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-cc4502be-0e92-4407-a801-44e88406e725 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 24 15:34:26.519032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:26.519032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 15:34:26.519032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 15:34:26.519032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 15:34:26.519032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 15:34:26.519032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 15:34:26.519032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:26.519032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 24 15:34:26.519032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:26.519032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 15:34:26.519032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:26.519032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:26.519032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:26.519032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:26.519032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:26.519032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:26.520170 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:26.520170 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:26.520170 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:26.520170 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:26.520170 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:26.520170 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:26.520170 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 15:34:26.520170 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 15:34:26.520170 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 15:34:26.520170 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 15:34:26.520170 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:26.520170 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:26.520170 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:26.520170 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:26.520170 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:26.520170 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:26.521017 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:26.521017 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:26.521017 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:26.521017 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:26.521017 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:26.521017 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:26.521017 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:26.521017 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:26.521017 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:26.521017 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:26.521017 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 15:34:26.521017 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:26.521017 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 24 15:34:26.521017 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:26.521017 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 410/847] 10.222.0.102 () {56 vars in 1119 bytes} [Wed Aug 24 15:34:26 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:26.521820 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4502be-0e92-4407-a801-44e88406e725 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:26.533948 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8cd091-aeb8-4ed4-af76-93a0365e2dc3 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:26.570320 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:26.571405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d092d64-c45c-498b-957f-e0b674e03dae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:26.572225 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d092d64-c45c-498b-957f-e0b674e03dae None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:26.572525 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d092d64-c45c-498b-957f-e0b674e03dae None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:26.573176 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d092d64-c45c-498b-957f-e0b674e03dae None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:26.639843 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4502be-0e92-4407-a801-44e88406e725 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:26.651881 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d092d64-c45c-498b-957f-e0b674e03dae None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:26.653262 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d092d64-c45c-498b-957f-e0b674e03dae None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:26.673075 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60769c77-122f-49f8-9fd7-231872eeac50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:26.673697 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60769c77-122f-49f8-9fd7-231872eeac50 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:26.673976 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60769c77-122f-49f8-9fd7-231872eeac50 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:26.674249 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60769c77-122f-49f8-9fd7-231872eeac50 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:26.687268 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60769c77-122f-49f8-9fd7-231872eeac50 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:26.688961 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60769c77-122f-49f8-9fd7-231872eeac50 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:26.706614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-60769c77-122f-49f8-9fd7-231872eeac50 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:26.706614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:26.706614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:26.706614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:26.706614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:26.706614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:26.706614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:26.706614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:26.706614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:26.706614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:26.706614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:26.706614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:26.706614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:26.706614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:26.706614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:26.706614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:26.707865 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:26.707865 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:26.707865 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:26.707865 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:26.707865 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:26.707865 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:26.707865 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:26.707865 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:26.707865 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:26.707865 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:26.707865 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:26.707865 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:26.707865 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:26.707865 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:26.707865 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:26.707865 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:26.707865 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:26.708963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:26.708963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:26.708963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:26.712806 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 411/848] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:26.713577 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60769c77-122f-49f8-9fd7-231872eeac50 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:26.759814 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:26.773811 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 438/849] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:34:26 2022] GET /identity/v3/auth/tokens => generated 1163 bytes in 361 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 15:34:26.774909 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d092d64-c45c-498b-957f-e0b674e03dae None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:26.831025 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:26.839388 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60769c77-122f-49f8-9fd7-231872eeac50 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:26.876978 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49cc33cb-beeb-4007-8717-a82fa740c74b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:26.877584 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49cc33cb-beeb-4007-8717-a82fa740c74b None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:26.877869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49cc33cb-beeb-4007-8717-a82fa740c74b None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:26.878144 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49cc33cb-beeb-4007-8717-a82fa740c74b None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:26.878782 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49cc33cb-beeb-4007-8717-a82fa740c74b None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:26.879087 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49cc33cb-beeb-4007-8717-a82fa740c74b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:26.880483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49cc33cb-beeb-4007-8717-a82fa740c74b None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:26.885026 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 412/850] 10.222.0.102 () {56 vars in 1119 bytes} [Wed Aug 24 15:34:26 2022] GET /identity/v3/services?name=volumev3 => generated 126 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:26.888834 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49cc33cb-beeb-4007-8717-a82fa740c74b None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:26.898007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d092d64-c45c-498b-957f-e0b674e03dae None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:26.947050 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fe0185-0a6b-43a3-b752-3e10d26739cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:26.947578 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fe0185-0a6b-43a3-b752-3e10d26739cd None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:26.947886 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fe0185-0a6b-43a3-b752-3e10d26739cd None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:26.948158 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fe0185-0a6b-43a3-b752-3e10d26739cd None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:26.949112 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6fe0185-0a6b-43a3-b752-3e10d26739cd None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:26.949291 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6fe0185-0a6b-43a3-b752-3e10d26739cd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:26.950623 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6fe0185-0a6b-43a3-b752-3e10d26739cd None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:26.967984 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 439/851] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:26 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:26.967984 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fe0185-0a6b-43a3-b752-3e10d26739cd None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:27.009888 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49cc33cb-beeb-4007-8717-a82fa740c74b None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:27.021550 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:27.050529 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0febf196-16bb-43e5-bdc6-461997f2c110 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:27.051225 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0febf196-16bb-43e5-bdc6-461997f2c110 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:27.051506 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0febf196-16bb-43e5-bdc6-461997f2c110 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:27.051808 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0febf196-16bb-43e5-bdc6-461997f2c110 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:27.063122 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0febf196-16bb-43e5-bdc6-461997f2c110 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:27.064870 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0febf196-16bb-43e5-bdc6-461997f2c110 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:27.080162 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-0febf196-16bb-43e5-bdc6-461997f2c110 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:27.080162 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:27.080162 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:27.080162 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:27.080162 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:27.080162 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:27.080162 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:27.080162 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:27.080162 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:27.080162 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:27.080162 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:27.080162 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:27.080162 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:27.080162 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:27.080162 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:27.080162 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:27.081159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:27.081159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:27.081159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:27.081159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:27.081159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:27.081159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:27.081159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:27.081159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:27.081159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:27.081159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:27.081159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:27.081159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:27.081159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:27.081159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:27.081159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:27.081159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:27.081159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:27.082035 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:27.082035 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:27.082035 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:27.082035 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 413/852] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:27.082691 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0febf196-16bb-43e5-bdc6-461997f2c110 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:27.091922 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fe0185-0a6b-43a3-b752-3e10d26739cd None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:27.097564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:27.130514 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e108125-ece1-4fc5-9645-1028748a162b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:27.131126 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e108125-ece1-4fc5-9645-1028748a162b None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:27.131407 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e108125-ece1-4fc5-9645-1028748a162b None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:27.131714 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e108125-ece1-4fc5-9645-1028748a162b None admin] PATH_INFO: `/v3/services` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:27.132371 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e108125-ece1-4fc5-9645-1028748a162b None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:27.132660 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e108125-ece1-4fc5-9645-1028748a162b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:27.134029 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e108125-ece1-4fc5-9645-1028748a162b None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:27.147674 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 440/853] 10.222.0.102 () {56 vars in 1119 bytes} [Wed Aug 24 15:34:26 2022] GET /identity/v3/services?type=volumev3 => generated 126 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:27.147674 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e108125-ece1-4fc5-9645-1028748a162b None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:27.253295 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0febf196-16bb-43e5-bdc6-461997f2c110 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:27.296892 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efccc69b-3d6b-4b1e-9e66-2c29bc0e55f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:27.297739 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e108125-ece1-4fc5-9645-1028748a162b None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:27.298694 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efccc69b-3d6b-4b1e-9e66-2c29bc0e55f3 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:27.299136 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efccc69b-3d6b-4b1e-9e66-2c29bc0e55f3 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:27.299693 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efccc69b-3d6b-4b1e-9e66-2c29bc0e55f3 None admin] PATH_INFO: `/v3/users/swift` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:27.309390 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efccc69b-3d6b-4b1e-9e66-2c29bc0e55f3 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:27.311143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efccc69b-3d6b-4b1e-9e66-2c29bc0e55f3 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:27.314914 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:27.329053 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-efccc69b-3d6b-4b1e-9e66-2c29bc0e55f3 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 24 15:34:27.329053 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:27.329053 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:27.329053 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:27.329053 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:27.329053 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:27.329053 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:27.329053 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:27.329053 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:27.329053 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:27.329053 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:27.329053 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:27.329053 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:34:27.329053 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:34:27.329053 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:34:27.329053 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:34:27.330197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:27.330197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:27.330197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:34:27.330197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:27.330197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:34:27.330197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:27.330197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:27.330197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:27.330197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:27.330197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:27.330197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:27.330197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:27.330197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:27.330197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:27.330197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:27.330197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:27.330197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:27.330197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:27.331195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:27.331195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:27.331195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:34:27.331195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:34:27.331195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:34:27.331195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:34:27.331195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:34:27.331195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:34:27.331195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 24 15:34:27.331195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:27.331195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 414/854] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:34:27 2022] GET /identity/v3/users/swift => generated 83 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:27.336775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efccc69b-3d6b-4b1e-9e66-2c29bc0e55f3 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:27.351946 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14944a50-70b3-428e-b8b2-b4fcbbd377a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:27.353032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14944a50-70b3-428e-b8b2-b4fcbbd377a3 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:27.353584 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14944a50-70b3-428e-b8b2-b4fcbbd377a3 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:27.354068 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14944a50-70b3-428e-b8b2-b4fcbbd377a3 None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:27.354954 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14944a50-70b3-428e-b8b2-b4fcbbd377a3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:27.355481 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14944a50-70b3-428e-b8b2-b4fcbbd377a3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:27.357165 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14944a50-70b3-428e-b8b2-b4fcbbd377a3 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:27.372227 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 441/855] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:27 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:27.378256 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14944a50-70b3-428e-b8b2-b4fcbbd377a3 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:27.428096 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:27.473055 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efccc69b-3d6b-4b1e-9e66-2c29bc0e55f3 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:27.495026 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14944a50-70b3-428e-b8b2-b4fcbbd377a3 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:27.518051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef407ea7-ee70-4546-884b-ad4075e30f77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:27.518627 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef407ea7-ee70-4546-884b-ad4075e30f77 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:27.518908 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef407ea7-ee70-4546-884b-ad4075e30f77 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:27.519181 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef407ea7-ee70-4546-884b-ad4075e30f77 None admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:27.519948 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef407ea7-ee70-4546-884b-ad4075e30f77 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:27.520252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef407ea7-ee70-4546-884b-ad4075e30f77 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:27.522461 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef407ea7-ee70-4546-884b-ad4075e30f77 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:27.542169 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa8b6cb-c63c-4fe9-afdf-e2c02e087adf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:27.542845 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa8b6cb-c63c-4fe9-afdf-e2c02e087adf None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:27.543799 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa8b6cb-c63c-4fe9-afdf-e2c02e087adf None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:27.544361 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa8b6cb-c63c-4fe9-afdf-e2c02e087adf None admin] PATH_INFO: `/v3/projects/service` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:27.547609 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 415/856] 10.222.0.102 () {56 vars in 1140 bytes} [Wed Aug 24 15:34:27 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 372 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:27.556667 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef407ea7-ee70-4546-884b-ad4075e30f77 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:27.556969 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa8b6cb-c63c-4fe9-afdf-e2c02e087adf None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:27.558561 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa8b6cb-c63c-4fe9-afdf-e2c02e087adf None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:27.573974 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-0aa8b6cb-c63c-4fe9-afdf-e2c02e087adf None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:27.573974 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:27.573974 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:27.573974 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:27.573974 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:27.573974 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:27.573974 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:27.573974 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:27.573974 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:27.573974 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:27.573974 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:27.573974 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:27.573974 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:34:27.573974 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:34:27.573974 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:34:27.574941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:34:27.574941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:27.574941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:27.574941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:27.574941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:27.574941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:27.574941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:27.574941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:27.574941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:27.574941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:27.574941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:27.574941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:27.574941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:27.574941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:27.574941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:27.574941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:27.574941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:27.576281 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:34:27.576281 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:34:27.576281 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:34:27.576281 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:34:27.576281 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:34:27.576281 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:34:27.576281 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:27.576281 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:27.579720 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 442/857] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:34:27 2022] GET /identity/v3/projects/service => generated 88 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:27.582331 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa8b6cb-c63c-4fe9-afdf-e2c02e087adf None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:27.620286 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:27.682669 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef407ea7-ee70-4546-884b-ad4075e30f77 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:27.712567 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa8b6cb-c63c-4fe9-afdf-e2c02e087adf None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:27.716564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:27.735484 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce5fcf2-8928-4abf-afca-530372c01891 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:27.736256 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce5fcf2-8928-4abf-afca-530372c01891 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:27.736743 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce5fcf2-8928-4abf-afca-530372c01891 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:27.737040 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce5fcf2-8928-4abf-afca-530372c01891 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:27.760323 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc26dfb2-49b2-4a9d-afe5-2b1c784e6956 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:27.761276 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc26dfb2-49b2-4a9d-afe5-2b1c784e6956 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:27.761558 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc26dfb2-49b2-4a9d-afe5-2b1c784e6956 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:27.761830 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc26dfb2-49b2-4a9d-afe5-2b1c784e6956 None admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:27.762523 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc26dfb2-49b2-4a9d-afe5-2b1c784e6956 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:27.762828 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc26dfb2-49b2-4a9d-afe5-2b1c784e6956 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:27.764295 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc26dfb2-49b2-4a9d-afe5-2b1c784e6956 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:27.780505 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 443/858] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:34:27 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:27.781864 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f925fb-e0c9-4c17-9b9f-210f682b6fa7 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:27.782147 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f925fb-e0c9-4c17-9b9f-210f682b6fa7 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:27.782415 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f925fb-e0c9-4c17-9b9f-210f682b6fa7 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:27.783675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 444/859] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:27 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:27.786472 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f925fb-e0c9-4c17-9b9f-210f682b6fa7 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:27.818307 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce5fcf2-8928-4abf-afca-530372c01891 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:27.819965 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce5fcf2-8928-4abf-afca-530372c01891 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:27.896159 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:27.908532 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f925fb-e0c9-4c17-9b9f-210f682b6fa7 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:27.935060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 416/860] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:34:27 2022] GET /identity/v3/auth/tokens => generated 1163 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 15:34:27.935060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282bd75f-48ba-48db-89a5-c6e0cb07bc16 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:27.935060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282bd75f-48ba-48db-89a5-c6e0cb07bc16 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:27.935060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282bd75f-48ba-48db-89a5-c6e0cb07bc16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:27.953013 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-282bd75f-48ba-48db-89a5-c6e0cb07bc16 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:27.954389 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-282bd75f-48ba-48db-89a5-c6e0cb07bc16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:27.961542 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ec0eaf-b038-4275-b85d-8aa660564241 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:27.962165 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ec0eaf-b038-4275-b85d-8aa660564241 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:27.962443 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ec0eaf-b038-4275-b85d-8aa660564241 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:27.962712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ec0eaf-b038-4275-b85d-8aa660564241 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:27.964092 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ec0eaf-b038-4275-b85d-8aa660564241 None admin] RBAC: Adding query filter params (role.id=628bab09d1dc48fd9e01023b2f7671f6, scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5, user.id=a6c17620ead14cb49d962990e1b1e0d8) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:27.964408 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ec0eaf-b038-4275-b85d-8aa660564241 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:27.965884 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ec0eaf-b038-4275-b85d-8aa660564241 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:27.973029 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 445/861] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:34:27 2022] GET /identity/v3/role_assignments?user.id=a6c17620ead14cb49d962990e1b1e0d8&scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5&role.id=628bab09d1dc48fd9e01023b2f7671f6 => generated 610 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:27.973904 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ec0eaf-b038-4275-b85d-8aa660564241 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:27.995655 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:28.058497 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 417/862] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:27 2022] POST /identity/v3/auth/tokens => generated 1163 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:34:28.106077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ec0eaf-b038-4275-b85d-8aa660564241 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:28.143925 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417e69a9-450c-47d5-aa21-5b0699f4837f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:28.145414 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417e69a9-450c-47d5-aa21-5b0699f4837f None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:28.145414 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417e69a9-450c-47d5-aa21-5b0699f4837f None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:28.145414 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417e69a9-450c-47d5-aa21-5b0699f4837f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:28.157462 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-417e69a9-450c-47d5-aa21-5b0699f4837f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:28.158928 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-417e69a9-450c-47d5-aa21-5b0699f4837f None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:28.175687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-417e69a9-450c-47d5-aa21-5b0699f4837f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:28.175687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:28.175687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:28.175687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:28.175687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:28.175687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:28.175687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:28.175687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:28.175687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:28.175687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:28.175687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:28.175687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:28.175687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:28.175687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:28.175687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:28.175687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:28.176712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:28.176712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:28.176712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:28.176712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:28.176712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:28.176712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:28.176712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:28.176712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:28.176712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:28.176712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:28.176712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:28.176712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:28.176712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:28.176712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:28.176712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:28.176712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:28.176712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:28.177625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:28.177625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:28.177625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:28.177625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 446/863] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:28.184199 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282bd75f-48ba-48db-89a5-c6e0cb07bc16 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:28.300149 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282bd75f-48ba-48db-89a5-c6e0cb07bc16 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:28.353349 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91422017-dd9a-462c-8659-e1c0710a0331 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:28.354260 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91422017-dd9a-462c-8659-e1c0710a0331 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:28.354719 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91422017-dd9a-462c-8659-e1c0710a0331 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:28.355176 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91422017-dd9a-462c-8659-e1c0710a0331 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:28.356057 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91422017-dd9a-462c-8659-e1c0710a0331 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:28.356548 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91422017-dd9a-462c-8659-e1c0710a0331 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:28.358824 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91422017-dd9a-462c-8659-e1c0710a0331 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:28.371734 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 418/864] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:28 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:28.378260 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417e69a9-450c-47d5-aa21-5b0699f4837f None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:28.381955 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91422017-dd9a-462c-8659-e1c0710a0331 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:28.395274 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:28.404278 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:28.502907 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91422017-dd9a-462c-8659-e1c0710a0331 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:28.517411 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417e69a9-450c-47d5-aa21-5b0699f4837f None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:28.540493 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860d32d5-2726-4780-ad91-c23ae506fc71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:28.541224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860d32d5-2726-4780-ad91-c23ae506fc71 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:28.541510 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860d32d5-2726-4780-ad91-c23ae506fc71 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:28.541837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860d32d5-2726-4780-ad91-c23ae506fc71 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:28.542428 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860d32d5-2726-4780-ad91-c23ae506fc71 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:28.544039 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860d32d5-2726-4780-ad91-c23ae506fc71 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:28.554042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-860d32d5-2726-4780-ad91-c23ae506fc71 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 24 15:34:28.554042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:28.554042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 15:34:28.554042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 15:34:28.554042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 15:34:28.554042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 15:34:28.554042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 15:34:28.554042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:28.554042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 24 15:34:28.554042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:28.554042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 15:34:28.554042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:28.554042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:28.554042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:28.554042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:28.554042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:28.555074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:28.555074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:28.555074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:28.555074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:28.555074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:28.555074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:28.555074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:28.555074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 15:34:28.555074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 15:34:28.555074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 15:34:28.555074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 15:34:28.555074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:28.555074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:28.555074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:28.555074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:28.555074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:28.555074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:28.556000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:28.556000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:28.556000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:28.556000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:28.556000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:28.556000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:28.556000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:28.556000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:28.556000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:28.556000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:28.556000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 15:34:28.556000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:28.556000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 24 15:34:28.556000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:28.556000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 419/865] 10.222.0.102 () {56 vars in 1137 bytes} [Wed Aug 24 15:34:28 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:28.561104 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860d32d5-2726-4780-ad91-c23ae506fc71 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:28.562139 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac8f77d-9cb8-4f8e-a103-b1fb7b514bd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:28.562783 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac8f77d-9cb8-4f8e-a103-b1fb7b514bd9 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:28.563058 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac8f77d-9cb8-4f8e-a103-b1fb7b514bd9 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:28.563326 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac8f77d-9cb8-4f8e-a103-b1fb7b514bd9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:28.574622 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac8f77d-9cb8-4f8e-a103-b1fb7b514bd9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:28.578062 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac8f77d-9cb8-4f8e-a103-b1fb7b514bd9 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:28.596866 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-dac8f77d-9cb8-4f8e-a103-b1fb7b514bd9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:28.596866 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:28.596866 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:28.596866 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:28.596866 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:28.596866 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:28.596866 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:28.596866 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:28.596866 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:28.596866 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:28.596866 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:28.596866 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:28.596866 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:34:28.596866 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:34:28.596866 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:34:28.597817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:34:28.597817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:28.597817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:28.597817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:28.597817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:28.597817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:28.597817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:28.597817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:28.597817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:28.597817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:28.597817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:28.597817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:28.597817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:28.597817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:28.597817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:28.597817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:28.597817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:28.598699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:34:28.598699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:34:28.598699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:34:28.598699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:34:28.598699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:34:28.598699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:34:28.598699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:28.598699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:28.602696 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 447/866] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:34:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:28.604991 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac8f77d-9cb8-4f8e-a103-b1fb7b514bd9 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:28.687806 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:28.717235 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860d32d5-2726-4780-ad91-c23ae506fc71 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:28.790958 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981435fa-6288-4a94-9b48-8c8d3ac79baa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:28.792042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981435fa-6288-4a94-9b48-8c8d3ac79baa None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:28.792530 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981435fa-6288-4a94-9b48-8c8d3ac79baa None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:28.793006 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981435fa-6288-4a94-9b48-8c8d3ac79baa None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:28.793866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-981435fa-6288-4a94-9b48-8c8d3ac79baa None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:28.795795 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-981435fa-6288-4a94-9b48-8c8d3ac79baa None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:28.799430 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-981435fa-6288-4a94-9b48-8c8d3ac79baa None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:28.829077 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 420/867] 10.222.0.102 () {56 vars in 1131 bytes} [Wed Aug 24 15:34:28 2022] GET /identity/v3/services?name=compute_legacy => generated 132 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:28.830393 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981435fa-6288-4a94-9b48-8c8d3ac79baa None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:28.844033 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:28.904212 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac8f77d-9cb8-4f8e-a103-b1fb7b514bd9 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:28.951198 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b598eb6-cd45-485b-82a8-ed7f95a52ba3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:28.951870 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b598eb6-cd45-485b-82a8-ed7f95a52ba3 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:28.955061 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b598eb6-cd45-485b-82a8-ed7f95a52ba3 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:28.955061 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b598eb6-cd45-485b-82a8-ed7f95a52ba3 None admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:28.955366 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b598eb6-cd45-485b-82a8-ed7f95a52ba3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:28.955706 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b598eb6-cd45-485b-82a8-ed7f95a52ba3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:28.957168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b598eb6-cd45-485b-82a8-ed7f95a52ba3 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:28.989101 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 448/868] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:34:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 385 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:28.997424 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b598eb6-cd45-485b-82a8-ed7f95a52ba3 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:29.059663 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981435fa-6288-4a94-9b48-8c8d3ac79baa None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:29.112230 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db0b623-18fe-4c9e-80fc-c0ad77d5211a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:29.112855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db0b623-18fe-4c9e-80fc-c0ad77d5211a None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:29.113141 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db0b623-18fe-4c9e-80fc-c0ad77d5211a None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:29.113423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db0b623-18fe-4c9e-80fc-c0ad77d5211a None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:29.114254 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db0b623-18fe-4c9e-80fc-c0ad77d5211a None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:29.114565 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db0b623-18fe-4c9e-80fc-c0ad77d5211a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:29.117022 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db0b623-18fe-4c9e-80fc-c0ad77d5211a None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:29.127975 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 421/869] 10.222.0.102 () {56 vars in 1131 bytes} [Wed Aug 24 15:34:28 2022] GET /identity/v3/services?type=compute_legacy => generated 132 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:29.161056 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b598eb6-cd45-485b-82a8-ed7f95a52ba3 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:29.193986 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:29.204826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e582ce-7dbd-41a0-bedd-e11b41c9f941 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:29.205448 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e582ce-7dbd-41a0-bedd-e11b41c9f941 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:29.205792 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e582ce-7dbd-41a0-bedd-e11b41c9f941 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:29.206063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e582ce-7dbd-41a0-bedd-e11b41c9f941 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:29.206905 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8e582ce-7dbd-41a0-bedd-e11b41c9f941 None admin] RBAC: Adding query filter params (role.id=4d8fbbaea81a4145b89af12350bb067c, scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5, user.id=6a070d4fd8c4466c857bd03965c11349) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:29.207223 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8e582ce-7dbd-41a0-bedd-e11b41c9f941 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:29.208774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8e582ce-7dbd-41a0-bedd-e11b41c9f941 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:29.216289 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 449/870] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:34:28 2022] GET /identity/v3/role_assignments?user.id=6a070d4fd8c4466c857bd03965c11349&scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5&role.id=4d8fbbaea81a4145b89af12350bb067c => generated 610 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:29.259294 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd20763-0a14-40c9-bb0e-5fd6ad98cb52 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:29.259579 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd20763-0a14-40c9-bb0e-5fd6ad98cb52 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:29.259902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd20763-0a14-40c9-bb0e-5fd6ad98cb52 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:29.261713 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 422/871] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:29 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:29.268985 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83287d93-5b44-49c2-86db-1a5d167ee71b None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:29.269263 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83287d93-5b44-49c2-86db-1a5d167ee71b None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:29.269536 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83287d93-5b44-49c2-86db-1a5d167ee71b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:29.291388 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-83287d93-5b44-49c2-86db-1a5d167ee71b None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:29.292734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-83287d93-5b44-49c2-86db-1a5d167ee71b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:29.380798 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 450/872] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:29 2022] POST /identity/v3/auth/tokens => generated 1163 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:34:29.652753 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd20763-0a14-40c9-bb0e-5fd6ad98cb52 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:29.666577 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:29.771343 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd20763-0a14-40c9-bb0e-5fd6ad98cb52 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:29.811375 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77577e80-bc1f-4bda-ab93-e9fc26536ebf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:29.812023 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77577e80-bc1f-4bda-ab93-e9fc26536ebf None admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:29.812427 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77577e80-bc1f-4bda-ab93-e9fc26536ebf None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:29.812773 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77577e80-bc1f-4bda-ab93-e9fc26536ebf None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:29.813448 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77577e80-bc1f-4bda-ab93-e9fc26536ebf None admin] RBAC: Authorizing `identity:create_service()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:29.815096 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77577e80-bc1f-4bda-ab93-e9fc26536ebf None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:29.835081 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 423/873] 10.222.0.102 () {60 vars in 1146 bytes} [Wed Aug 24 15:34:29 2022] POST /identity/v3/services => generated 252 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:34:30.249550 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d23245f-6a9f-43c3-93f8-d62c6e2bfcb1 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:30.249954 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d23245f-6a9f-43c3-93f8-d62c6e2bfcb1 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:30.250231 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d23245f-6a9f-43c3-93f8-d62c6e2bfcb1 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:30.251839 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 451/874] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:30 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:30.257417 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f9a845-fbe6-430f-8096-bd4158bffe4e None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:30.257740 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f9a845-fbe6-430f-8096-bd4158bffe4e None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:30.257984 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f9a845-fbe6-430f-8096-bd4158bffe4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:30.277246 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-48f9a845-fbe6-430f-8096-bd4158bffe4e None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:30.278454 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-48f9a845-fbe6-430f-8096-bd4158bffe4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:30.354011 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:30.368008 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 424/875] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:30 2022] POST /identity/v3/auth/tokens => generated 1264 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:34:30.637307 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d23245f-6a9f-43c3-93f8-d62c6e2bfcb1 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:30.754921 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d23245f-6a9f-43c3-93f8-d62c6e2bfcb1 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:30.779930 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:30.792698 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe72917e-4dad-4028-b4bf-3f567d81c3e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:30.793337 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe72917e-4dad-4028-b4bf-3f567d81c3e5 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:30.793757 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe72917e-4dad-4028-b4bf-3f567d81c3e5 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:30.794228 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe72917e-4dad-4028-b4bf-3f567d81c3e5 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:30.805922 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe72917e-4dad-4028-b4bf-3f567d81c3e5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:34:30.810645 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-fe72917e-4dad-4028-b4bf-3f567d81c3e5 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 24 15:34:30.810645 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:30.810645 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:30.810645 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:30.810645 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:30.810645 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:30.810645 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:30.810645 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:30.810645 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:30.810645 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:30.810645 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:30.810645 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:30.810645 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:34:30.810645 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:34:30.810645 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:34:30.810645 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:34:30.812127 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:34:30.812127 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:34:30.812127 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:34:30.812127 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:34:30.812127 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:30.812127 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:30.812127 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:30.812127 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:30.812127 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:30.812127 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:30.812127 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:30.812127 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:30.812127 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:30.812127 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:30.812127 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:30.812127 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:30.814166 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:30.814166 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:30.814166 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:30.814166 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:30.814166 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:34:30.814166 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:34:30.814166 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:34:30.814166 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:34:30.814166 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:34:30.814166 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:34:30.814166 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 24 15:34:30.814166 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:30.814166 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 452/876] 10.222.0.102 () {56 vars in 1125 bytes} [Wed Aug 24 15:34:30 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:30.818489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f9a845-fbe6-430f-8096-bd4158bffe4e None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:30.934316 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f9a845-fbe6-430f-8096-bd4158bffe4e None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:30.973394 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260e6cf6-79cd-4b66-b48d-615dddad0c2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:30.974016 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260e6cf6-79cd-4b66-b48d-615dddad0c2f None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:30.974299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260e6cf6-79cd-4b66-b48d-615dddad0c2f None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:30.974575 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260e6cf6-79cd-4b66-b48d-615dddad0c2f None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:30.975269 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260e6cf6-79cd-4b66-b48d-615dddad0c2f None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:30.975579 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260e6cf6-79cd-4b66-b48d-615dddad0c2f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:30.977028 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260e6cf6-79cd-4b66-b48d-615dddad0c2f None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:30.990106 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 425/877] 10.222.0.102 () {56 vars in 1120 bytes} [Wed Aug 24 15:34:30 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 340 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:30.995492 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe72917e-4dad-4028-b4bf-3f567d81c3e5 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:31.111696 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe72917e-4dad-4028-b4bf-3f567d81c3e5 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:31.151682 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c57f075-9baf-4633-8670-ba3d01fcc5b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:31.152328 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c57f075-9baf-4633-8670-ba3d01fcc5b0 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:31.152618 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c57f075-9baf-4633-8670-ba3d01fcc5b0 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:31.152904 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c57f075-9baf-4633-8670-ba3d01fcc5b0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:31.164755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c57f075-9baf-4633-8670-ba3d01fcc5b0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:31.166222 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c57f075-9baf-4633-8670-ba3d01fcc5b0 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:31.182337 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-1c57f075-9baf-4633-8670-ba3d01fcc5b0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:31.182337 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:31.182337 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:31.182337 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:31.182337 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:31.182337 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:31.182337 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:31.182337 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:31.182337 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:31.182337 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:31.182337 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:31.182337 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:31.182337 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:31.182337 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:31.182337 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:31.182337 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:31.183324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:31.183324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:31.183324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:31.183324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:31.183324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:31.183324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:31.183324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:31.183324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:31.183324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:31.183324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:31.183324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:31.183324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:31.183324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:31.183324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:31.183324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:31.183324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:31.183324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:31.184250 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:31.184250 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:31.184250 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:31.184250 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 453/878] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:31.191270 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260e6cf6-79cd-4b66-b48d-615dddad0c2f None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:31.243675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082ac17a-ec16-4378-8dc6-2897601c9c8e None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:31.243675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082ac17a-ec16-4378-8dc6-2897601c9c8e None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:31.243675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082ac17a-ec16-4378-8dc6-2897601c9c8e None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:31.243675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 454/879] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:31 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:31.247877 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8039910b-172b-41fa-8c93-330889d50045 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:31.248162 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8039910b-172b-41fa-8c93-330889d50045 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:31.248436 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8039910b-172b-41fa-8c93-330889d50045 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:31.261490 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:31.274463 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-8039910b-172b-41fa-8c93-330889d50045 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:31.276020 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-8039910b-172b-41fa-8c93-330889d50045 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:31.307899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:31.321169 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260e6cf6-79cd-4b66-b48d-615dddad0c2f None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:31.360296 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 455/880] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:31 2022] POST /identity/v3/auth/tokens => generated 1264 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 15:34:31.365608 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133ed431-7a31-4a2a-a109-6de59c338156 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:31.365902 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133ed431-7a31-4a2a-a109-6de59c338156 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:31.366145 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133ed431-7a31-4a2a-a109-6de59c338156 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:31.371970 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 456/881] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:31 2022] GET /identity => generated 271 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:31.375695 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c52640-ac6d-4889-8861-006f05fcf3ad None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:31.376282 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c52640-ac6d-4889-8861-006f05fcf3ad None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:31.376772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c52640-ac6d-4889-8861-006f05fcf3ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:31.381121 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc06f5f-5cff-4b41-bded-cd199a46ab34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:31.381729 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc06f5f-5cff-4b41-bded-cd199a46ab34 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:31.382011 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc06f5f-5cff-4b41-bded-cd199a46ab34 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:31.382355 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc06f5f-5cff-4b41-bded-cd199a46ab34 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:31.383015 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc06f5f-5cff-4b41-bded-cd199a46ab34 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:31.383322 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc06f5f-5cff-4b41-bded-cd199a46ab34 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:31.384744 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc06f5f-5cff-4b41-bded-cd199a46ab34 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:31.399511 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-e4c52640-ac6d-4889-8861-006f05fcf3ad None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:31.401147 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-e4c52640-ac6d-4889-8861-006f05fcf3ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:31.404028 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 426/882] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:31 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:31.408812 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc06f5f-5cff-4b41-bded-cd199a46ab34 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:31.478196 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 457/883] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:31 2022] POST /identity/v3/auth/tokens => generated 1264 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:34:31.528368 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc06f5f-5cff-4b41-bded-cd199a46ab34 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:31.548354 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:31.572076 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da86e65-acf4-47e5-a088-04c5ea2c7e64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:31.573192 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da86e65-acf4-47e5-a088-04c5ea2c7e64 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:31.573689 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da86e65-acf4-47e5-a088-04c5ea2c7e64 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:31.574179 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da86e65-acf4-47e5-a088-04c5ea2c7e64 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:31.582462 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da86e65-acf4-47e5-a088-04c5ea2c7e64 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:31.584359 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da86e65-acf4-47e5-a088-04c5ea2c7e64 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:31.598709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-4da86e65-acf4-47e5-a088-04c5ea2c7e64 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 24 15:34:31.598709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:31.598709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:31.598709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:31.598709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:31.598709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:31.598709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:31.598709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:31.598709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:31.598709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:31.598709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:31.598709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:31.598709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:34:31.598709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:34:31.598709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:34:31.598709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:34:31.599738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:31.599738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:31.599738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:34:31.599738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:31.599738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:34:31.599738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:31.599738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:31.599738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:31.599738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:31.599738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:31.599738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:31.599738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:31.599738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:31.599738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:31.599738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:31.599738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:31.599738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:31.599738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:31.600679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:31.600679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:31.600679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:34:31.600679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:34:31.600679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:34:31.600679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:34:31.600679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:34:31.600679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:34:31.600679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 24 15:34:31.600679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:31.603205 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 427/884] 10.222.0.102 () {56 vars in 1122 bytes} [Wed Aug 24 15:34:31 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:34:31.609441 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c52640-ac6d-4889-8861-006f05fcf3ad None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:31.632358 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da86e65-acf4-47e5-a088-04c5ea2c7e64 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:31.724796 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:31.735504 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c52640-ac6d-4889-8861-006f05fcf3ad None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:31.764379 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da86e65-acf4-47e5-a088-04c5ea2c7e64 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:31.774307 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aceefafd-b621-4c65-bae7-13ab4fc99e5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:31.775279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aceefafd-b621-4c65-bae7-13ab4fc99e5b None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:31.775796 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aceefafd-b621-4c65-bae7-13ab4fc99e5b None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:31.776274 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aceefafd-b621-4c65-bae7-13ab4fc99e5b None admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:31.777209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aceefafd-b621-4c65-bae7-13ab4fc99e5b None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:31.777766 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aceefafd-b621-4c65-bae7-13ab4fc99e5b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:31.779415 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aceefafd-b621-4c65-bae7-13ab4fc99e5b None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:31.799798 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 458/885] 10.222.0.102 () {56 vars in 1154 bytes} [Wed Aug 24 15:34:31 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 386 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:31.800714 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aceefafd-b621-4c65-bae7-13ab4fc99e5b None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:31.809873 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16417b34-77cc-494c-8422-5075bcd65a79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:31.810486 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16417b34-77cc-494c-8422-5075bcd65a79 None admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:31.810767 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16417b34-77cc-494c-8422-5075bcd65a79 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:31.811041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16417b34-77cc-494c-8422-5075bcd65a79 None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:31.812418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16417b34-77cc-494c-8422-5075bcd65a79 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:31.814032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16417b34-77cc-494c-8422-5075bcd65a79 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:31.837871 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 428/886] 10.222.0.102 () {60 vars in 1146 bytes} [Wed Aug 24 15:34:31 2022] POST /identity/v3/services => generated 270 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 15:34:31.845126 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16417b34-77cc-494c-8422-5075bcd65a79 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:31.897736 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:31.965519 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aceefafd-b621-4c65-bae7-13ab4fc99e5b None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:31.991970 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16417b34-77cc-494c-8422-5075bcd65a79 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:32.010629 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb779ba-0edb-48de-9aba-3d9106776846 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:32.011277 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb779ba-0edb-48de-9aba-3d9106776846 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:32.011822 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb779ba-0edb-48de-9aba-3d9106776846 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:32.011822 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb779ba-0edb-48de-9aba-3d9106776846 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:32.012426 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb779ba-0edb-48de-9aba-3d9106776846 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:32.013805 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb779ba-0edb-48de-9aba-3d9106776846 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:32.026032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-bdb779ba-0edb-48de-9aba-3d9106776846 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 24 15:34:32.026032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:32.026032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 15:34:32.026032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 15:34:32.026032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 15:34:32.026032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 15:34:32.026032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 15:34:32.026032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:32.026032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 24 15:34:32.026032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:32.026032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 15:34:32.026032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:32.026032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:32.026032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:32.026032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:32.026032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:32.026032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:32.027107 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:32.027107 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:32.027107 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:32.027107 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:32.027107 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:32.027107 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:32.027107 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 15:34:32.027107 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 15:34:32.027107 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 15:34:32.027107 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 15:34:32.027107 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:32.027107 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:32.027107 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:32.027107 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:32.027107 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:32.027107 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:32.028045 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:32.028045 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:32.028045 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:32.028045 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:32.028045 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:32.028045 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:32.028045 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:32.028045 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:32.028045 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:32.028045 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:32.028045 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 15:34:32.028045 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:32.028045 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 24 15:34:32.028045 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:32.028045 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 459/887] 10.222.0.102 () {56 vars in 1131 bytes} [Wed Aug 24 15:34:31 2022] GET /identity/v3/services/object-store => generated 93 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:32.028822 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b77177-a359-43ed-902c-8f4785d84737 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:32.028822 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b77177-a359-43ed-902c-8f4785d84737 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:32.029057 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b77177-a359-43ed-902c-8f4785d84737 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:32.031012 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 460/888] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:32 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:32.037339 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6650f557-62e1-4e91-8344-a52185d04f11 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:32.037538 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6650f557-62e1-4e91-8344-a52185d04f11 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:32.038003 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6650f557-62e1-4e91-8344-a52185d04f11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:32.042932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd6bdde-0868-40f7-b179-bb1a8b93a455 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:32.043541 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd6bdde-0868-40f7-b179-bb1a8b93a455 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:32.043852 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd6bdde-0868-40f7-b179-bb1a8b93a455 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:32.044128 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd6bdde-0868-40f7-b179-bb1a8b93a455 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:32.056671 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-6650f557-62e1-4e91-8344-a52185d04f11 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:32.057873 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-6650f557-62e1-4e91-8344-a52185d04f11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:32.070625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:32.120050 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd6bdde-0868-40f7-b179-bb1a8b93a455 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:32.121542 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd6bdde-0868-40f7-b179-bb1a8b93a455 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:32.137211 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 461/889] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:32 2022] POST /identity/v3/auth/tokens => generated 1374 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:34:32.138428 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6650f557-62e1-4e91-8344-a52185d04f11 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:32.166504 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:32.255216 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 429/890] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:34:31 2022] GET /identity/v3/auth/tokens => generated 1374 bytes in 411 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 15:34:32.261191 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd6bdde-0868-40f7-b179-bb1a8b93a455 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:32.287363 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6650f557-62e1-4e91-8344-a52185d04f11 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:32.327002 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c325785a-3dfa-4f72-80f4-117a6a9fa99f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:32.328585 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c325785a-3dfa-4f72-80f4-117a6a9fa99f None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:32.328931 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c325785a-3dfa-4f72-80f4-117a6a9fa99f None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:32.329213 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c325785a-3dfa-4f72-80f4-117a6a9fa99f None admin] PATH_INFO: `/v3/services` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:32.329852 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c325785a-3dfa-4f72-80f4-117a6a9fa99f None admin] RBAC: Adding query filter params (name=object-store) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:32.330372 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c325785a-3dfa-4f72-80f4-117a6a9fa99f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:32.332385 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c325785a-3dfa-4f72-80f4-117a6a9fa99f None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:32.343452 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 462/891] 10.222.0.102 () {56 vars in 1127 bytes} [Wed Aug 24 15:34:32 2022] GET /identity/v3/services?name=object-store => generated 130 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:32.351658 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c325785a-3dfa-4f72-80f4-117a6a9fa99f None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:32.360348 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:32.374735 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd6bdde-0868-40f7-b179-bb1a8b93a455 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:32.415772 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78ad842-1c4f-47b5-bcef-ff7654538207 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:32.416499 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78ad842-1c4f-47b5-bcef-ff7654538207 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:32.416818 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78ad842-1c4f-47b5-bcef-ff7654538207 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:32.417294 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78ad842-1c4f-47b5-bcef-ff7654538207 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:32.420204 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:32.431968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a78ad842-1c4f-47b5-bcef-ff7654538207 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:32.433432 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a78ad842-1c4f-47b5-bcef-ff7654538207 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:32.449319 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-a78ad842-1c4f-47b5-bcef-ff7654538207 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:32.449319 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:32.449319 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:32.449319 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:32.449319 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:32.449319 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:32.449319 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:32.449319 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:32.449319 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:32.449319 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:32.449319 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:32.449319 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:32.449319 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:32.449319 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:32.449319 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:32.449319 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:32.450324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:32.450324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:32.450324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:32.450324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:32.450324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:32.450324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:32.450324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:32.450324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:32.450324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:32.450324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:32.450324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:32.450324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:32.450324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:32.450324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:32.450324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:32.450324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:32.450324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:32.451226 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:32.451226 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:32.451226 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:32.451226 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 430/892] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:34:32.451226 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78ad842-1c4f-47b5-bcef-ff7654538207 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:32.473683 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c325785a-3dfa-4f72-80f4-117a6a9fa99f None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:32.511340 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75faaf00-b54c-48f8-83b4-729cf95971e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:32.511888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75faaf00-b54c-48f8-83b4-729cf95971e4 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:32.512174 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75faaf00-b54c-48f8-83b4-729cf95971e4 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:32.512447 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75faaf00-b54c-48f8-83b4-729cf95971e4 None admin] PATH_INFO: `/v3/services` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:32.513190 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75faaf00-b54c-48f8-83b4-729cf95971e4 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:32.513500 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75faaf00-b54c-48f8-83b4-729cf95971e4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:32.515762 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75faaf00-b54c-48f8-83b4-729cf95971e4 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:32.528047 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 463/893] 10.222.0.102 () {56 vars in 1127 bytes} [Wed Aug 24 15:34:32 2022] GET /identity/v3/services?type=object-store => generated 130 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:32.529371 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75faaf00-b54c-48f8-83b4-729cf95971e4 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:32.565557 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78ad842-1c4f-47b5-bcef-ff7654538207 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:32.613854 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d068148d-cb8d-402e-aadd-2e385a01a89d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:32.615127 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d068148d-cb8d-402e-aadd-2e385a01a89d None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:32.615594 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d068148d-cb8d-402e-aadd-2e385a01a89d None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:32.616313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d068148d-cb8d-402e-aadd-2e385a01a89d None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:32.617611 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d068148d-cb8d-402e-aadd-2e385a01a89d None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:32.619805 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d068148d-cb8d-402e-aadd-2e385a01a89d None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:32.634406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-d068148d-cb8d-402e-aadd-2e385a01a89d None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 24 15:34:32.634406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:32.634406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 15:34:32.634406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 15:34:32.634406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 15:34:32.634406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 15:34:32.634406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 15:34:32.634406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:32.634406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 24 15:34:32.634406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:32.634406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 15:34:32.634406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:32.634406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:32.634406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:32.634406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:32.634406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:32.634406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:32.635464 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:32.635464 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:32.635464 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:32.635464 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:32.635464 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:32.635464 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:32.635464 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 15:34:32.635464 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 15:34:32.635464 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 15:34:32.635464 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 15:34:32.635464 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:32.635464 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:32.635464 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:32.635464 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:32.635464 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:32.635464 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:32.636338 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:32.636338 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:32.636338 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:32.636338 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:32.636338 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:32.636338 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:32.636338 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:32.636338 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:32.636338 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:32.636338 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:32.636338 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 15:34:32.636338 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:32.636338 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 24 15:34:32.636338 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:32.639354 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 431/894] 10.222.0.102 () {56 vars in 1119 bytes} [Wed Aug 24 15:34:32 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:32.644759 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:32.646262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d068148d-cb8d-402e-aadd-2e385a01a89d None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:32.691236 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75faaf00-b54c-48f8-83b4-729cf95971e4 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:32.719894 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:32.739395 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6356602-4989-4c97-9707-3d42b4242b3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:32.740657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6356602-4989-4c97-9707-3d42b4242b3d None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:32.740937 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6356602-4989-4c97-9707-3d42b4242b3d None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:32.741210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6356602-4989-4c97-9707-3d42b4242b3d None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:32.742183 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6356602-4989-4c97-9707-3d42b4242b3d None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:32.742495 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6356602-4989-4c97-9707-3d42b4242b3d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:32.745089 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6356602-4989-4c97-9707-3d42b4242b3d None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:32.761021 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 464/895] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:32 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:32.767389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6356602-4989-4c97-9707-3d42b4242b3d None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:32.777581 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d068148d-cb8d-402e-aadd-2e385a01a89d None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:32.815230 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36dfc5cc-981f-4b65-8fd3-ccbffa8c08fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:32.815868 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dfc5cc-981f-4b65-8fd3-ccbffa8c08fb None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:32.816148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dfc5cc-981f-4b65-8fd3-ccbffa8c08fb None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:32.816422 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dfc5cc-981f-4b65-8fd3-ccbffa8c08fb None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:32.817067 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36dfc5cc-981f-4b65-8fd3-ccbffa8c08fb None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:32.817371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36dfc5cc-981f-4b65-8fd3-ccbffa8c08fb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:32.818846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36dfc5cc-981f-4b65-8fd3-ccbffa8c08fb None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:32.829374 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 432/896] 10.222.0.102 () {56 vars in 1119 bytes} [Wed Aug 24 15:34:32 2022] GET /identity/v3/services?name=volumev3 => generated 126 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:32.830607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36dfc5cc-981f-4b65-8fd3-ccbffa8c08fb None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:32.910972 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6356602-4989-4c97-9707-3d42b4242b3d None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:32.968316 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d0b115-4319-462f-986d-65465f345caf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:32.969162 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d0b115-4319-462f-986d-65465f345caf None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:32.969538 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d0b115-4319-462f-986d-65465f345caf None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:32.969905 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d0b115-4319-462f-986d-65465f345caf None admin] PATH_INFO: `/v3/projects/service` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:32.972340 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:32.976109 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36dfc5cc-981f-4b65-8fd3-ccbffa8c08fb None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:32.979541 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d0b115-4319-462f-986d-65465f345caf None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:32.981167 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d0b115-4319-462f-986d-65465f345caf None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:32.995893 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-a4d0b115-4319-462f-986d-65465f345caf None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:32.995893 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:32.995893 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:32.995893 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:32.995893 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:32.995893 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:32.995893 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:32.995893 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:32.995893 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:32.995893 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:32.995893 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:32.995893 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:32.995893 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:34:32.995893 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:34:32.995893 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:34:32.997088 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:34:32.997088 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:32.997088 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:32.997088 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:32.997088 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:32.997088 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:32.997088 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:32.997088 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:32.997088 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:32.997088 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:32.997088 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:32.997088 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:32.997088 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:32.997088 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:32.997088 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:32.997088 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:32.997088 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:32.998209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:34:32.998209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:34:32.998209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:34:32.998209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:34:32.998209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:34:32.998209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:34:32.998209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:32.998209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:32.999900 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 465/897] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:34:32 2022] GET /identity/v3/projects/service => generated 88 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:33.000792 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:33.005379 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d0b115-4319-462f-986d-65465f345caf None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:33.026285 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009e5930-637c-4b02-ae07-6acbc3641c08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:33.027578 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009e5930-637c-4b02-ae07-6acbc3641c08 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:33.028156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009e5930-637c-4b02-ae07-6acbc3641c08 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:33.028941 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009e5930-637c-4b02-ae07-6acbc3641c08 None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:33.030971 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009e5930-637c-4b02-ae07-6acbc3641c08 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:33.031597 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009e5930-637c-4b02-ae07-6acbc3641c08 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:33.033331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009e5930-637c-4b02-ae07-6acbc3641c08 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:33.047529 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 433/898] 10.222.0.102 () {56 vars in 1119 bytes} [Wed Aug 24 15:34:32 2022] GET /identity/v3/services?type=volumev3 => generated 364 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:33.048512 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009e5930-637c-4b02-ae07-6acbc3641c08 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:33.145908 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d0b115-4319-462f-986d-65465f345caf None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:33.161573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009e5930-637c-4b02-ae07-6acbc3641c08 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:33.182706 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c899de5-c93b-42e3-8c8d-1ce9cb51f208 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:33.183585 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c899de5-c93b-42e3-8c8d-1ce9cb51f208 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:33.184157 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c899de5-c93b-42e3-8c8d-1ce9cb51f208 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:33.184600 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c899de5-c93b-42e3-8c8d-1ce9cb51f208 None admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:33.185441 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c899de5-c93b-42e3-8c8d-1ce9cb51f208 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:33.185893 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c899de5-c93b-42e3-8c8d-1ce9cb51f208 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:33.188806 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c899de5-c93b-42e3-8c8d-1ce9cb51f208 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:33.198697 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf17e52-ac48-4f1c-91c8-f1a769e9d4cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:33.199334 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf17e52-ac48-4f1c-91c8-f1a769e9d4cb None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:33.199926 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf17e52-ac48-4f1c-91c8-f1a769e9d4cb None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:33.200042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf17e52-ac48-4f1c-91c8-f1a769e9d4cb None admin] PATH_INFO: `/v3/endpoints` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:33.200764 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf17e52-ac48-4f1c-91c8-f1a769e9d4cb None admin] RBAC: Adding query filter params (interface=public, service_id=ddc7efef664e4b3e9cddc4f0e1622ab8, region_id=RegionOne) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:33.201088 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf17e52-ac48-4f1c-91c8-f1a769e9d4cb None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:33.203198 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf17e52-ac48-4f1c-91c8-f1a769e9d4cb None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:33.204732 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 466/899] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:34:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:33.211718 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c899de5-c93b-42e3-8c8d-1ce9cb51f208 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:33.219271 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 434/900] 10.222.0.102 () {56 vars in 1256 bytes} [Wed Aug 24 15:34:33 2022] GET /identity/v3/endpoints?service_id=ddc7efef664e4b3e9cddc4f0e1622ab8&interface=public®ion_id=RegionOne => generated 195 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:33.220539 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf17e52-ac48-4f1c-91c8-f1a769e9d4cb None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:33.285465 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:33.289869 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:33.347979 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c899de5-c93b-42e3-8c8d-1ce9cb51f208 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:33.379766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf17e52-ac48-4f1c-91c8-f1a769e9d4cb None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:33.386666 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a295cde-99cc-4534-849b-5aaef9bd131d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:33.387624 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a295cde-99cc-4534-849b-5aaef9bd131d None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:33.388137 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a295cde-99cc-4534-849b-5aaef9bd131d None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:33.388609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a295cde-99cc-4534-849b-5aaef9bd131d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:33.389566 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a295cde-99cc-4534-849b-5aaef9bd131d None admin] RBAC: Adding query filter params (role.id=5cd803f4cb3d42f0b728362ffd2f43cd, scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5, user.id=a6c17620ead14cb49d962990e1b1e0d8) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:33.390952 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a295cde-99cc-4534-849b-5aaef9bd131d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:33.392750 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a295cde-99cc-4534-849b-5aaef9bd131d None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:33.399506 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 467/901] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:34:33 2022] GET /identity/v3/role_assignments?user.id=a6c17620ead14cb49d962990e1b1e0d8&scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5&role.id=5cd803f4cb3d42f0b728362ffd2f43cd => generated 260 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:33.424003 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c565a9-d172-4cf2-9950-b260933a1c36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:33.425048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c565a9-d172-4cf2-9950-b260933a1c36 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:33.425533 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c565a9-d172-4cf2-9950-b260933a1c36 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:33.426067 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c565a9-d172-4cf2-9950-b260933a1c36 None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:33.426985 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c565a9-d172-4cf2-9950-b260933a1c36 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:33.428721 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c565a9-d172-4cf2-9950-b260933a1c36 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:33.436418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 435/902] 10.222.0.102 () {56 vars in 1092 bytes} [Wed Aug 24 15:34:33 2022] GET /identity/v3/services => generated 1274 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:34:33.928594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80edec4b-7c36-47c9-af33-59ed4d8baf19 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:33.929570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80edec4b-7c36-47c9-af33-59ed4d8baf19 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:33.930049 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80edec4b-7c36-47c9-af33-59ed4d8baf19 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:33.931965 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 468/903] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:33 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:33.938582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9db24c-5f71-427e-8602-706132bea9bf None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:33.939203 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9db24c-5f71-427e-8602-706132bea9bf None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:33.939718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9db24c-5f71-427e-8602-706132bea9bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:33.960308 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-4e9db24c-5f71-427e-8602-706132bea9bf None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:33.961857 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-4e9db24c-5f71-427e-8602-706132bea9bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:34.037627 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 436/904] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:33 2022] POST /identity/v3/auth/tokens => generated 1374 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:34:34.308901 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80edec4b-7c36-47c9-af33-59ed4d8baf19 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:34.419895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80edec4b-7c36-47c9-af33-59ed4d8baf19 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:34.439185 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:34.457103 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22412221-5cbf-475f-8220-67817cde7243 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:34.457745 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22412221-5cbf-475f-8220-67817cde7243 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:34.458028 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22412221-5cbf-475f-8220-67817cde7243 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:34.458305 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22412221-5cbf-475f-8220-67817cde7243 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:34.458898 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22412221-5cbf-475f-8220-67817cde7243 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:34.460306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22412221-5cbf-475f-8220-67817cde7243 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:34.470357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-22412221-5cbf-475f-8220-67817cde7243 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 24 15:34:34.470357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:34.470357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 15:34:34.470357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 15:34:34.470357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 15:34:34.470357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 15:34:34.470357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 15:34:34.470357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:34.470357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 24 15:34:34.470357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:34.470357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 15:34:34.470357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:34.470357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:34.470357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:34.470357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:34.470357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:34.471369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:34.471369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:34.471369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:34.471369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:34.471369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:34.471369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:34.471369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:34.471369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 15:34:34.471369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 15:34:34.471369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 15:34:34.471369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 15:34:34.471369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:34.471369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:34.471369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:34.471369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:34.471369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:34.471369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:34.472307 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:34.472307 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:34.472307 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:34.472307 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:34.472307 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:34.472307 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:34.472307 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:34.472307 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:34.472307 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:34.472307 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:34.472307 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 15:34:34.472307 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:34.472307 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 24 15:34:34.472307 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:34.472307 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 469/905] 10.222.0.102 () {56 vars in 1137 bytes} [Wed Aug 24 15:34:34 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:34.479696 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9db24c-5f71-427e-8602-706132bea9bf None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:34.520948 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:34.590545 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9db24c-5f71-427e-8602-706132bea9bf None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:34.623006 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835807c7-9029-4c27-bf48-218e7d7ce8c3 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:34.623357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835807c7-9029-4c27-bf48-218e7d7ce8c3 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:34.623602 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835807c7-9029-4c27-bf48-218e7d7ce8c3 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:34.629716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd05fc3-699c-4311-8700-8e58f37bdcd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:34.629980 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 470/906] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:34 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:34:34.630782 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd05fc3-699c-4311-8700-8e58f37bdcd5 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:34.631262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd05fc3-699c-4311-8700-8e58f37bdcd5 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:34.631473 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e889401-3406-4626-81e6-4d21839900d1 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:34.631744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e889401-3406-4626-81e6-4d21839900d1 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:34.632029 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e889401-3406-4626-81e6-4d21839900d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:34.632365 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd05fc3-699c-4311-8700-8e58f37bdcd5 None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:34.633192 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd05fc3-699c-4311-8700-8e58f37bdcd5 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:34.633711 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd05fc3-699c-4311-8700-8e58f37bdcd5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:34.635763 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd05fc3-699c-4311-8700-8e58f37bdcd5 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:34.642104 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 437/907] 10.222.0.102 () {56 vars in 1131 bytes} [Wed Aug 24 15:34:34 2022] GET /identity/v3/services?name=compute_legacy => generated 132 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:34.648333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd05fc3-699c-4311-8700-8e58f37bdcd5 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:34.652071 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-7e889401-3406-4626-81e6-4d21839900d1 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:34.653550 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-7e889401-3406-4626-81e6-4d21839900d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:34.743353 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 471/908] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:34 2022] POST /identity/v3/auth/tokens => generated 1374 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 15:34:34.771813 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd05fc3-699c-4311-8700-8e58f37bdcd5 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:34.776054 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:34.808030 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1f0f7c-1eae-4573-a1d2-658f97a45f88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:34.808702 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1f0f7c-1eae-4573-a1d2-658f97a45f88 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:34.809045 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1f0f7c-1eae-4573-a1d2-658f97a45f88 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:34.809320 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1f0f7c-1eae-4573-a1d2-658f97a45f88 None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:34.810014 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1f0f7c-1eae-4573-a1d2-658f97a45f88 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:34.810440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1f0f7c-1eae-4573-a1d2-658f97a45f88 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:34.812611 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1f0f7c-1eae-4573-a1d2-658f97a45f88 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:34.818289 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 438/909] 10.222.0.102 () {56 vars in 1131 bytes} [Wed Aug 24 15:34:34 2022] GET /identity/v3/services?type=compute_legacy => generated 388 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:34.824096 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e889401-3406-4626-81e6-4d21839900d1 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:34.935471 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e889401-3406-4626-81e6-4d21839900d1 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:34.951301 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:34.977254 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46aa6301-eca1-407c-8cf6-67be05f76b40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:34.977878 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46aa6301-eca1-407c-8cf6-67be05f76b40 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:34.978158 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46aa6301-eca1-407c-8cf6-67be05f76b40 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:34.978430 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46aa6301-eca1-407c-8cf6-67be05f76b40 None admin] PATH_INFO: `/v3/endpoints` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:34.979158 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46aa6301-eca1-407c-8cf6-67be05f76b40 None admin] RBAC: Adding query filter params (interface=public, service_id=12c98bf3431041fd948ba93c310e65d9, region_id=RegionOne) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:34.979466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46aa6301-eca1-407c-8cf6-67be05f76b40 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:34.980874 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46aa6301-eca1-407c-8cf6-67be05f76b40 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:34.987938 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 472/910] 10.222.0.102 () {56 vars in 1256 bytes} [Wed Aug 24 15:34:34 2022] GET /identity/v3/endpoints?service_id=12c98bf3431041fd948ba93c310e65d9&interface=public®ion_id=RegionOne => generated 195 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:34.993689 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1f0f7c-1eae-4573-a1d2-658f97a45f88 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:35.041920 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46aa6301-eca1-407c-8cf6-67be05f76b40 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:35.123157 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1f0f7c-1eae-4573-a1d2-658f97a45f88 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:35.159555 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46aa6301-eca1-407c-8cf6-67be05f76b40 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:35.167678 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215452f5-b41f-45ee-8b3f-b95b7987a04d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:35.168862 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215452f5-b41f-45ee-8b3f-b95b7987a04d None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:35.169150 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215452f5-b41f-45ee-8b3f-b95b7987a04d None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:35.169428 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215452f5-b41f-45ee-8b3f-b95b7987a04d None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:35.170049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215452f5-b41f-45ee-8b3f-b95b7987a04d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:35.172872 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215452f5-b41f-45ee-8b3f-b95b7987a04d None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:35.188585 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 439/911] 10.222.0.102 () {56 vars in 1092 bytes} [Wed Aug 24 15:34:34 2022] GET /identity/v3/services => generated 1274 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:34:35.206619 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8305b4e-31aa-48f8-a69e-141edf18a1e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:35.207719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8305b4e-31aa-48f8-a69e-141edf18a1e4 None admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:35.208233 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8305b4e-31aa-48f8-a69e-141edf18a1e4 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:35.208706 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8305b4e-31aa-48f8-a69e-141edf18a1e4 None admin] PATH_INFO: `/v3/services` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:35.209651 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8305b4e-31aa-48f8-a69e-141edf18a1e4 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:35.211413 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8305b4e-31aa-48f8-a69e-141edf18a1e4 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:35.235923 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 473/912] 10.222.0.102 () {60 vars in 1146 bytes} [Wed Aug 24 15:34:35 2022] POST /identity/v3/services => generated 242 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:34:35.352258 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2355bcdd-c634-4714-ab07-7f7b0c3782d9 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:35.352600 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2355bcdd-c634-4714-ab07-7f7b0c3782d9 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:35.352814 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2355bcdd-c634-4714-ab07-7f7b0c3782d9 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:35.354478 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 440/913] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:35 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:35.360951 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e8db8a-ae94-4233-b44a-1dc7649d6069 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:35.361263 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e8db8a-ae94-4233-b44a-1dc7649d6069 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:35.361507 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e8db8a-ae94-4233-b44a-1dc7649d6069 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:35.383474 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-b4e8db8a-ae94-4233-b44a-1dc7649d6069 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:35.384742 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-b4e8db8a-ae94-4233-b44a-1dc7649d6069 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:35.419718 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:35.471468 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 474/914] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:35 2022] POST /identity/v3/auth/tokens => generated 1476 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:34:35.498779 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b397ecd9-2ef1-403e-9c30-81ca3e771a84 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:35.499947 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b397ecd9-2ef1-403e-9c30-81ca3e771a84 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:35.500245 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b397ecd9-2ef1-403e-9c30-81ca3e771a84 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:35.501592 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 441/915] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:35 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:35.507282 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2ca405-dc85-4470-a443-e47649902f29 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:35.507568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2ca405-dc85-4470-a443-e47649902f29 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:35.507875 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2ca405-dc85-4470-a443-e47649902f29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:35.527938 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-cd2ca405-dc85-4470-a443-e47649902f29 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:35.529137 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-cd2ca405-dc85-4470-a443-e47649902f29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:35.602958 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 475/916] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:35 2022] POST /identity/v3/auth/tokens => generated 1476 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:34:35.766392 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b397ecd9-2ef1-403e-9c30-81ca3e771a84 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:35.800018 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:35.891785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2ca405-dc85-4470-a443-e47649902f29 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:35.894747 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b397ecd9-2ef1-403e-9c30-81ca3e771a84 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:35.936635 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9004d7b2-279f-4bc4-ace6-563842b00fac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:35.938121 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9004d7b2-279f-4bc4-ace6-563842b00fac None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:35.938410 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9004d7b2-279f-4bc4-ace6-563842b00fac None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:35.938941 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9004d7b2-279f-4bc4-ace6-563842b00fac None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:35.950442 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9004d7b2-279f-4bc4-ace6-563842b00fac None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:34:35.956817 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-9004d7b2-279f-4bc4-ace6-563842b00fac None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 24 15:34:35.956817 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:35.956817 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:35.956817 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:35.956817 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:35.956817 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:35.956817 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:35.956817 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:35.956817 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:35.956817 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:35.956817 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:35.956817 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:35.956817 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:34:35.956817 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:34:35.956817 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:34:35.956817 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:34:35.957886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:34:35.957886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:34:35.957886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:34:35.957886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:34:35.957886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:35.957886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:35.957886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:35.957886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:35.957886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:35.957886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:35.957886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:35.957886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:35.957886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:35.957886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:35.957886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:35.957886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:35.958704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:35.958704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:35.958704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:35.958704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:35.958704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:34:35.958704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:34:35.958704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:34:35.958704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:34:35.958704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:34:35.958704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:34:35.958704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 24 15:34:35.958704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:35.958704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 442/917] 10.222.0.102 () {56 vars in 1125 bytes} [Wed Aug 24 15:34:35 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:35.964395 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9004d7b2-279f-4bc4-ace6-563842b00fac None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:36.010347 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:36.022524 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2ca405-dc85-4470-a443-e47649902f29 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:36.060726 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7fb013-11ba-4772-ab5e-5f1b7a013a3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:36.061365 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7fb013-11ba-4772-ab5e-5f1b7a013a3e None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:36.062216 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7fb013-11ba-4772-ab5e-5f1b7a013a3e None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:36.062820 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7fb013-11ba-4772-ab5e-5f1b7a013a3e None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:36.063841 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7fb013-11ba-4772-ab5e-5f1b7a013a3e None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:36.065501 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7fb013-11ba-4772-ab5e-5f1b7a013a3e None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:36.083338 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-4a7fb013-11ba-4772-ab5e-5f1b7a013a3e None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 24 15:34:36.083338 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:36.083338 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 15:34:36.083338 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 15:34:36.083338 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 15:34:36.083338 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 15:34:36.083338 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 15:34:36.083338 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:36.083338 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 24 15:34:36.083338 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:36.083338 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 15:34:36.083338 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:36.083338 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:36.083338 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:36.083338 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:36.083338 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:36.083338 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:36.084405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:36.084405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:36.084405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:36.084405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:36.084405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:36.084405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:36.084405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 15:34:36.084405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 15:34:36.084405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 15:34:36.084405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 15:34:36.084405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:36.084405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:36.084405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:36.084405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:36.084405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:36.084405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:36.085282 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:36.085282 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:36.085282 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:36.085282 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:36.085282 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:36.085282 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:36.085282 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:36.085282 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:36.085282 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:36.085282 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:36.085282 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 15:34:36.085282 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:36.085282 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 24 15:34:36.085282 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:36.092008 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 476/918] 10.222.0.102 () {56 vars in 1119 bytes} [Wed Aug 24 15:34:35 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:36.093966 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:36.095445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7fb013-11ba-4772-ab5e-5f1b7a013a3e None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:36.105484 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9004d7b2-279f-4bc4-ace6-563842b00fac None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:36.159902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e51f37-b596-4d18-8c12-eb6bc6f56421 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:36.160569 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e51f37-b596-4d18-8c12-eb6bc6f56421 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:36.160808 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e51f37-b596-4d18-8c12-eb6bc6f56421 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:36.161105 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e51f37-b596-4d18-8c12-eb6bc6f56421 None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:36.161782 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e51f37-b596-4d18-8c12-eb6bc6f56421 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:36.162140 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e51f37-b596-4d18-8c12-eb6bc6f56421 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:36.164171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e51f37-b596-4d18-8c12-eb6bc6f56421 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:36.179797 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 443/919] 10.222.0.102 () {56 vars in 1120 bytes} [Wed Aug 24 15:34:35 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 340 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:36.183226 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e51f37-b596-4d18-8c12-eb6bc6f56421 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:36.225811 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7fb013-11ba-4772-ab5e-5f1b7a013a3e None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:36.271449 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4779b58e-c936-4a70-a8a9-1d5c039251d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:36.272509 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4779b58e-c936-4a70-a8a9-1d5c039251d7 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:36.273069 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4779b58e-c936-4a70-a8a9-1d5c039251d7 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:36.273535 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4779b58e-c936-4a70-a8a9-1d5c039251d7 None admin] PATH_INFO: `/v3/services` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:36.274410 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4779b58e-c936-4a70-a8a9-1d5c039251d7 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:36.274915 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4779b58e-c936-4a70-a8a9-1d5c039251d7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:36.276592 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4779b58e-c936-4a70-a8a9-1d5c039251d7 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:36.283451 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 477/920] 10.222.0.102 () {56 vars in 1119 bytes} [Wed Aug 24 15:34:36 2022] GET /identity/v3/services?name=volumev3 => generated 126 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:36.289692 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4779b58e-c936-4a70-a8a9-1d5c039251d7 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:36.316815 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:36.343160 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e51f37-b596-4d18-8c12-eb6bc6f56421 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:36.396144 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c63203c-5f15-491b-88bc-43fb8b6f6c46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:36.397847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c63203c-5f15-491b-88bc-43fb8b6f6c46 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:36.398378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c63203c-5f15-491b-88bc-43fb8b6f6c46 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:36.399146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c63203c-5f15-491b-88bc-43fb8b6f6c46 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:36.416170 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c63203c-5f15-491b-88bc-43fb8b6f6c46 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:36.418822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c63203c-5f15-491b-88bc-43fb8b6f6c46 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:36.419804 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4779b58e-c936-4a70-a8a9-1d5c039251d7 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:36.437290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-7c63203c-5f15-491b-88bc-43fb8b6f6c46 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:36.437290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:36.437290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:36.437290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:36.437290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:36.437290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:36.437290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:36.437290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:36.437290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:36.437290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:36.437290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:36.437290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:36.437290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:36.437290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:36.437290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:36.437290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:36.438846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:36.438846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:36.438846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:36.438846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:36.438846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:36.438846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:36.438846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:36.438846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:36.438846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:36.438846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:36.438846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:36.438846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:36.438846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:36.438846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:36.438846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:36.438846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:36.438846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:36.439786 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:36.439786 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:36.439786 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:36.443549 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 444/921] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:36.444782 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c63203c-5f15-491b-88bc-43fb8b6f6c46 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:36.447580 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:36.471381 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e8ace4-87b0-401d-bcfb-82b5537892ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:36.472077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e8ace4-87b0-401d-bcfb-82b5537892ff None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:36.472359 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e8ace4-87b0-401d-bcfb-82b5537892ff None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:36.472636 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e8ace4-87b0-401d-bcfb-82b5537892ff None admin] PATH_INFO: `/v3/services` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:36.473698 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e8ace4-87b0-401d-bcfb-82b5537892ff None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:36.474019 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e8ace4-87b0-401d-bcfb-82b5537892ff None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:36.475487 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e8ace4-87b0-401d-bcfb-82b5537892ff None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:36.484385 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 478/922] 10.222.0.102 () {56 vars in 1119 bytes} [Wed Aug 24 15:34:36 2022] GET /identity/v3/services?type=volumev3 => generated 364 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:36.490213 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e8ace4-87b0-401d-bcfb-82b5537892ff None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:36.588674 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c63203c-5f15-491b-88bc-43fb8b6f6c46 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:36.618367 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:36.628000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29015c8c-0724-447b-9247-c7ce0b4a7a89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:36.628744 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29015c8c-0724-447b-9247-c7ce0b4a7a89 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:36.629126 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29015c8c-0724-447b-9247-c7ce0b4a7a89 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:36.629498 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29015c8c-0724-447b-9247-c7ce0b4a7a89 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:36.630277 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29015c8c-0724-447b-9247-c7ce0b4a7a89 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:36.630758 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29015c8c-0724-447b-9247-c7ce0b4a7a89 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:36.632359 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29015c8c-0724-447b-9247-c7ce0b4a7a89 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:36.637556 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e8ace4-87b0-401d-bcfb-82b5537892ff None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:36.650243 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 445/923] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:36 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:36.652073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29015c8c-0724-447b-9247-c7ce0b4a7a89 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:36.682768 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c059f8b-c493-4bc4-977d-cee5d635478c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:36.683738 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c059f8b-c493-4bc4-977d-cee5d635478c None admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:36.684094 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c059f8b-c493-4bc4-977d-cee5d635478c None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:36.684440 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c059f8b-c493-4bc4-977d-cee5d635478c None admin] PATH_INFO: `/v3/endpoints` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:36.685553 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c059f8b-c493-4bc4-977d-cee5d635478c None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:36.688210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c059f8b-c493-4bc4-977d-cee5d635478c None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:36.719238 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 479/924] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:34:36 2022] POST /identity/v3/endpoints => generated 348 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 15:34:36.733347 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:36.782187 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29015c8c-0724-447b-9247-c7ce0b4a7a89 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:36.832693 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae856dd9-73ba-4fec-a46b-19b8f3374396 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:36.833692 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae856dd9-73ba-4fec-a46b-19b8f3374396 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:36.834147 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae856dd9-73ba-4fec-a46b-19b8f3374396 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:36.834653 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae856dd9-73ba-4fec-a46b-19b8f3374396 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:36.844053 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae856dd9-73ba-4fec-a46b-19b8f3374396 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:36.846037 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae856dd9-73ba-4fec-a46b-19b8f3374396 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:36.860042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-ae856dd9-73ba-4fec-a46b-19b8f3374396 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 24 15:34:36.860042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:36.860042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:36.860042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:36.860042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:36.860042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:36.860042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:36.860042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:36.860042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:36.860042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:36.860042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:36.860042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:36.860042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:34:36.860042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:34:36.860042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:34:36.860042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:34:36.861156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:36.861156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:36.861156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:34:36.861156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:36.861156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:34:36.861156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:36.861156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:36.861156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:36.861156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:36.861156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:36.861156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:36.861156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:36.861156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:36.861156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:36.861156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:36.861156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:36.861156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:36.861156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:36.862076 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:36.862076 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:36.862076 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:34:36.862076 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:34:36.862076 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:34:36.862076 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:34:36.862076 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:34:36.862076 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:34:36.862076 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 24 15:34:36.862076 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:36.866429 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 446/925] 10.222.0.102 () {56 vars in 1122 bytes} [Wed Aug 24 15:34:36 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:34:36.867954 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c059f8b-c493-4bc4-977d-cee5d635478c None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:36.984288 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c059f8b-c493-4bc4-977d-cee5d635478c None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:37.021982 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111b75b1-59cc-400d-956b-2aab84992b8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:37.022755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111b75b1-59cc-400d-956b-2aab84992b8e None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:37.022918 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111b75b1-59cc-400d-956b-2aab84992b8e None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:37.023162 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111b75b1-59cc-400d-956b-2aab84992b8e None admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:37.023940 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111b75b1-59cc-400d-956b-2aab84992b8e None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:37.024254 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111b75b1-59cc-400d-956b-2aab84992b8e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:37.025709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111b75b1-59cc-400d-956b-2aab84992b8e None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:37.037166 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:37.048574 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 480/926] 10.222.0.102 () {56 vars in 1154 bytes} [Wed Aug 24 15:34:36 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 386 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:37.055590 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae856dd9-73ba-4fec-a46b-19b8f3374396 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:37.169561 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae856dd9-73ba-4fec-a46b-19b8f3374396 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:37.190714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:37.209748 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b403bf67-cd05-4b1d-baa2-fd4b1d7e0981 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:37.210462 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b403bf67-cd05-4b1d-baa2-fd4b1d7e0981 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:37.210748 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b403bf67-cd05-4b1d-baa2-fd4b1d7e0981 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:37.211020 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b403bf67-cd05-4b1d-baa2-fd4b1d7e0981 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:37.223204 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b403bf67-cd05-4b1d-baa2-fd4b1d7e0981 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:37.224701 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b403bf67-cd05-4b1d-baa2-fd4b1d7e0981 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:37.239352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-b403bf67-cd05-4b1d-baa2-fd4b1d7e0981 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:37.239352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:37.239352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:37.239352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:37.239352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:37.239352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:37.239352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:37.239352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:37.239352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:37.239352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:37.239352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:37.239352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:37.239352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:37.239352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:37.239352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:37.239352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:37.240436 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:37.240436 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:37.240436 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:37.240436 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:37.240436 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:37.240436 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:37.240436 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:37.240436 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:37.240436 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:37.240436 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:37.240436 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:37.240436 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:37.240436 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:37.240436 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:37.240436 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:37.240436 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:37.240436 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:37.241646 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:37.241646 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:37.241646 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:37.241646 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 447/927] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:37.248786 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111b75b1-59cc-400d-956b-2aab84992b8e None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:37.305932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a9dbad-65e4-4e8c-a503-432a66d14c28 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:37.306284 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a9dbad-65e4-4e8c-a503-432a66d14c28 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:37.306502 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a9dbad-65e4-4e8c-a503-432a66d14c28 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:37.308174 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 448/928] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:37 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:37.313832 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010e6728-0bc8-41ea-b166-b5ab705831a9 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:37.314124 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010e6728-0bc8-41ea-b166-b5ab705831a9 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:37.314406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010e6728-0bc8-41ea-b166-b5ab705831a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:37.338282 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-010e6728-0bc8-41ea-b166-b5ab705831a9 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:37.339497 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-010e6728-0bc8-41ea-b166-b5ab705831a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:37.374424 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111b75b1-59cc-400d-956b-2aab84992b8e None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:37.417949 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f1feef-76c0-46b4-9836-027a17daa2fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:37.418570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f1feef-76c0-46b4-9836-027a17daa2fe None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:37.418875 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f1feef-76c0-46b4-9836-027a17daa2fe None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:37.419114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f1feef-76c0-46b4-9836-027a17daa2fe None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:37.419853 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f1feef-76c0-46b4-9836-027a17daa2fe None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:37.420102 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f1feef-76c0-46b4-9836-027a17daa2fe None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:37.421520 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f1feef-76c0-46b4-9836-027a17daa2fe None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:37.427351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 449/929] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:37 2022] POST /identity/v3/auth/tokens => generated 1476 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 15:34:37.428605 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad74de03-397f-485d-b095-fc6d7b35312b None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:37.428892 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad74de03-397f-485d-b095-fc6d7b35312b None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:37.429167 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad74de03-397f-485d-b095-fc6d7b35312b None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:37.430509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 450/930] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:37 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:37.436258 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84434dff-b7d3-4bc2-a3e2-3ce60d11991a None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:37.436490 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84434dff-b7d3-4bc2-a3e2-3ce60d11991a None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:37.436810 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84434dff-b7d3-4bc2-a3e2-3ce60d11991a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:37.439581 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 481/931] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:37 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:37.445197 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f1feef-76c0-46b4-9836-027a17daa2fe None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:37.464832 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-84434dff-b7d3-4bc2-a3e2-3ce60d11991a None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:37.466365 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-84434dff-b7d3-4bc2-a3e2-3ce60d11991a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:37.523164 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:37.560302 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 451/932] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:37 2022] POST /identity/v3/auth/tokens => generated 1476 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:34:37.575526 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f1feef-76c0-46b4-9836-027a17daa2fe None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:37.614136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f07929-a3c4-41fc-a209-67f45fd677aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:37.615287 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f07929-a3c4-41fc-a209-67f45fd677aa None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:37.615855 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f07929-a3c4-41fc-a209-67f45fd677aa None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:37.616338 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f07929-a3c4-41fc-a209-67f45fd677aa None admin] PATH_INFO: `/v3/projects/service` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:37.625871 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f07929-a3c4-41fc-a209-67f45fd677aa None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:37.627728 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f07929-a3c4-41fc-a209-67f45fd677aa None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:37.639688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-89f07929-a3c4-41fc-a209-67f45fd677aa None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:37.639688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:37.639688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:37.639688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:37.639688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:37.639688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:37.639688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:37.639688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:37.639688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:37.639688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:37.639688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:37.639688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:37.639688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:34:37.639688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:34:37.639688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:34:37.641613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:34:37.641613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:37.641613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:37.641613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:37.641613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:37.641613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:37.641613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:37.641613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:37.641613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:37.641613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:37.641613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:37.641613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:37.641613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:37.641613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:37.641613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:37.641613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:37.641613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:37.642845 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:34:37.642845 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:34:37.642845 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:34:37.642845 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:34:37.642845 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:34:37.642845 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:34:37.642845 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:37.642845 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:37.645604 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 482/933] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:34:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:37.651480 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84434dff-b7d3-4bc2-a3e2-3ce60d11991a None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:37.684121 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:37.714591 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f07929-a3c4-41fc-a209-67f45fd677aa None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:37.776011 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84434dff-b7d3-4bc2-a3e2-3ce60d11991a None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:37.832116 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec00ae8-b426-4aad-841f-c0a40cd1d3c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:37.833383 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec00ae8-b426-4aad-841f-c0a40cd1d3c5 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:37.833683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec00ae8-b426-4aad-841f-c0a40cd1d3c5 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:37.833960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec00ae8-b426-4aad-841f-c0a40cd1d3c5 None admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:37.834948 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec00ae8-b426-4aad-841f-c0a40cd1d3c5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:37.835262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec00ae8-b426-4aad-841f-c0a40cd1d3c5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:37.837445 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec00ae8-b426-4aad-841f-c0a40cd1d3c5 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:37.839864 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f07929-a3c4-41fc-a209-67f45fd677aa None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:37.857679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 452/934] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:34:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:37.859307 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec00ae8-b426-4aad-841f-c0a40cd1d3c5 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:37.867722 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:37.885537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3ad952-93b5-45b2-be77-0340322d1895 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:37.886179 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3ad952-93b5-45b2-be77-0340322d1895 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:37.886461 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3ad952-93b5-45b2-be77-0340322d1895 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:37.886734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3ad952-93b5-45b2-be77-0340322d1895 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:37.887353 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3ad952-93b5-45b2-be77-0340322d1895 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:37.888731 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3ad952-93b5-45b2-be77-0340322d1895 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:37.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-eb3ad952-93b5-45b2-be77-0340322d1895 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 24 15:34:37.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:37.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 15:34:37.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 15:34:37.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 15:34:37.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 15:34:37.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 15:34:37.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:37.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 24 15:34:37.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:37.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 15:34:37.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:37.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:37.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:37.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:37.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:37.899831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:37.899831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:37.899831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:37.899831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:37.899831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:37.899831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:37.899831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:37.899831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 15:34:37.899831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 15:34:37.899831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 15:34:37.899831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 15:34:37.899831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:37.899831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:37.899831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:37.899831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:37.899831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:37.899831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:37.904589 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:37.904589 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:37.904589 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:37.904589 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:37.904589 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:37.904589 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:37.904589 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:37.904589 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:37.904589 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:37.904589 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:37.904589 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 15:34:37.904589 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:37.904589 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 24 15:34:37.904589 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:37.904589 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 483/935] 10.222.0.102 () {56 vars in 1137 bytes} [Wed Aug 24 15:34:37 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:37.905434 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3ad952-93b5-45b2-be77-0340322d1895 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:37.978281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:37.989609 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec00ae8-b426-4aad-841f-c0a40cd1d3c5 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:38.025534 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6dae207-c5b0-4a5b-838b-c6097dac2f67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:38.026172 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6dae207-c5b0-4a5b-838b-c6097dac2f67 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:38.026453 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6dae207-c5b0-4a5b-838b-c6097dac2f67 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:38.026726 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6dae207-c5b0-4a5b-838b-c6097dac2f67 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:38.027341 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6dae207-c5b0-4a5b-838b-c6097dac2f67 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:38.028723 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6dae207-c5b0-4a5b-838b-c6097dac2f67 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:38.035048 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3ad952-93b5-45b2-be77-0340322d1895 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:38.038241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-e6dae207-c5b0-4a5b-838b-c6097dac2f67 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 24 15:34:38.038241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:38.038241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 15:34:38.038241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 15:34:38.038241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 15:34:38.038241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 15:34:38.038241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 15:34:38.038241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:38.038241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 24 15:34:38.038241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:38.038241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 15:34:38.038241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:38.038241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:38.038241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:38.038241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:38.038241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:38.038241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:38.045505 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:38.045505 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:38.045505 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:38.045505 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:38.045505 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:38.045505 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:38.045505 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 15:34:38.045505 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 15:34:38.045505 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 15:34:38.045505 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 15:34:38.045505 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:38.045505 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:38.045505 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:38.045505 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:38.045505 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:38.045505 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:38.046658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:38.046658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:38.046658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:38.046658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:38.046658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:38.046658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:38.046658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:38.046658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:38.046658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:38.046658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:38.046658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 15:34:38.046658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:38.046658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 24 15:34:38.046658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:38.046658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 453/936] 10.222.0.102 () {56 vars in 1131 bytes} [Wed Aug 24 15:34:37 2022] GET /identity/v3/services/object-store => generated 93 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:38.047475 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6dae207-c5b0-4a5b-838b-c6097dac2f67 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:38.081205 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6e9261-a644-49f0-973d-b03235c0f9e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:38.081877 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6e9261-a644-49f0-973d-b03235c0f9e4 None admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:38.082157 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6e9261-a644-49f0-973d-b03235c0f9e4 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:38.082434 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6e9261-a644-49f0-973d-b03235c0f9e4 None admin] PATH_INFO: `/v3/projects/3e50e6f6c1f44c648337c8d37a8e94a5/users/a6c17620ead14cb49d962990e1b1e0d8/roles/5cd803f4cb3d42f0b728362ffd2f43cd` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:38.084998 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6e9261-a644-49f0-973d-b03235c0f9e4 None admin] RBAC: Authorizing `identity:create_grant(project_id=3e50e6f6c1f44c648337c8d37a8e94a5, user_id=a6c17620ead14cb49d962990e1b1e0d8, role_id=5cd803f4cb3d42f0b728362ffd2f43cd)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:38.086447 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6e9261-a644-49f0-973d-b03235c0f9e4 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:38.101573 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 484/937] 10.222.0.102 () {58 vars in 1444 bytes} [Wed Aug 24 15:34:37 2022] PUT /identity/v3/projects/3e50e6f6c1f44c648337c8d37a8e94a5/users/a6c17620ead14cb49d962990e1b1e0d8/roles/5cd803f4cb3d42f0b728362ffd2f43cd => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:34:38.109262 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6e9261-a644-49f0-973d-b03235c0f9e4 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:38.161881 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:38.179652 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6dae207-c5b0-4a5b-838b-c6097dac2f67 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:38.223047 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde42d28-b8e1-4c42-8cb2-b923ba141a2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:38.224263 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde42d28-b8e1-4c42-8cb2-b923ba141a2e None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:38.224749 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde42d28-b8e1-4c42-8cb2-b923ba141a2e None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:38.225288 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde42d28-b8e1-4c42-8cb2-b923ba141a2e None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:38.226166 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde42d28-b8e1-4c42-8cb2-b923ba141a2e None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:38.226688 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde42d28-b8e1-4c42-8cb2-b923ba141a2e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:38.228643 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde42d28-b8e1-4c42-8cb2-b923ba141a2e None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:38.231311 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6e9261-a644-49f0-973d-b03235c0f9e4 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:38.241050 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 454/938] 10.222.0.102 () {56 vars in 1131 bytes} [Wed Aug 24 15:34:38 2022] GET /identity/v3/services?name=compute_legacy => generated 132 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:38.242348 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde42d28-b8e1-4c42-8cb2-b923ba141a2e None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:38.266098 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:38.274996 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa2bce4-0279-4a01-bf59-e4c728608a8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:38.275796 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa2bce4-0279-4a01-bf59-e4c728608a8e None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:38.275973 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa2bce4-0279-4a01-bf59-e4c728608a8e None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:38.276251 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa2bce4-0279-4a01-bf59-e4c728608a8e None admin] PATH_INFO: `/v3/services` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:38.276906 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa2bce4-0279-4a01-bf59-e4c728608a8e None admin] RBAC: Adding query filter params (name=object-store) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:38.277213 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa2bce4-0279-4a01-bf59-e4c728608a8e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:38.278615 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa2bce4-0279-4a01-bf59-e4c728608a8e None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:38.286500 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 485/939] 10.222.0.102 () {56 vars in 1127 bytes} [Wed Aug 24 15:34:38 2022] GET /identity/v3/services?name=object-store => generated 130 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:38.293258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa2bce4-0279-4a01-bf59-e4c728608a8e None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:38.370473 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde42d28-b8e1-4c42-8cb2-b923ba141a2e None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:38.407719 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cac8a81-3be5-459f-a264-8d544219120c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:38.408330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cac8a81-3be5-459f-a264-8d544219120c None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:38.408606 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cac8a81-3be5-459f-a264-8d544219120c None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:38.408890 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cac8a81-3be5-459f-a264-8d544219120c None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:38.409501 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cac8a81-3be5-459f-a264-8d544219120c None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:38.409802 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cac8a81-3be5-459f-a264-8d544219120c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:38.411180 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cac8a81-3be5-459f-a264-8d544219120c None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:38.416746 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 455/940] 10.222.0.102 () {56 vars in 1131 bytes} [Wed Aug 24 15:34:38 2022] GET /identity/v3/services?type=compute_legacy => generated 388 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:38.422385 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cac8a81-3be5-459f-a264-8d544219120c None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:38.437824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa2bce4-0279-4a01-bf59-e4c728608a8e None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:38.446918 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:38.479319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b96f10-6817-4abb-a2b0-46e1665d1812 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:38.480295 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b96f10-6817-4abb-a2b0-46e1665d1812 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:38.480887 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b96f10-6817-4abb-a2b0-46e1665d1812 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:38.481403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b96f10-6817-4abb-a2b0-46e1665d1812 None admin] PATH_INFO: `/v3/services` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:38.482308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34b96f10-6817-4abb-a2b0-46e1665d1812 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:38.482811 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34b96f10-6817-4abb-a2b0-46e1665d1812 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:38.484955 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34b96f10-6817-4abb-a2b0-46e1665d1812 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:38.496098 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 486/941] 10.222.0.102 () {56 vars in 1127 bytes} [Wed Aug 24 15:34:38 2022] GET /identity/v3/services?type=object-store => generated 358 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:38.497904 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b96f10-6817-4abb-a2b0-46e1665d1812 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:38.523605 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:38.544339 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cac8a81-3be5-459f-a264-8d544219120c None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:38.583338 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f3708a-a3eb-4962-bc67-ee631d563c69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:38.583550 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f3708a-a3eb-4962-bc67-ee631d563c69 None admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:38.583840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f3708a-a3eb-4962-bc67-ee631d563c69 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:38.584022 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f3708a-a3eb-4962-bc67-ee631d563c69 None admin] PATH_INFO: `/v3/endpoints` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:38.584788 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f3708a-a3eb-4962-bc67-ee631d563c69 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:38.586210 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f3708a-a3eb-4962-bc67-ee631d563c69 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:38.613362 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 456/942] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:34:38 2022] POST /identity/v3/endpoints => generated 349 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 15:34:38.627178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b96f10-6817-4abb-a2b0-46e1665d1812 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:38.682861 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2050b360-7f73-426d-8745-f0538ed6c3b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:38.683554 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2050b360-7f73-426d-8745-f0538ed6c3b3 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:38.683988 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2050b360-7f73-426d-8745-f0538ed6c3b3 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:38.684268 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2050b360-7f73-426d-8745-f0538ed6c3b3 None admin] PATH_INFO: `/v3/endpoints` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:38.685112 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2050b360-7f73-426d-8745-f0538ed6c3b3 None admin] RBAC: Adding query filter params (interface=public, service_id=71e4b2caecb345f694d8317e607bfa38, region_id=RegionOne) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:38.685429 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2050b360-7f73-426d-8745-f0538ed6c3b3 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:38.687796 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2050b360-7f73-426d-8745-f0538ed6c3b3 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:38.696445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 487/943] 10.222.0.102 () {56 vars in 1256 bytes} [Wed Aug 24 15:34:38 2022] GET /identity/v3/endpoints?service_id=71e4b2caecb345f694d8317e607bfa38&interface=public®ion_id=RegionOne => generated 195 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:38.704868 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f3708a-a3eb-4962-bc67-ee631d563c69 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:38.828046 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f3708a-a3eb-4962-bc67-ee631d563c69 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:38.868231 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908ca27d-a65c-4fe1-8587-c25fe1939350 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:38.868840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908ca27d-a65c-4fe1-8587-c25fe1939350 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:38.869116 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908ca27d-a65c-4fe1-8587-c25fe1939350 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:38.869447 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908ca27d-a65c-4fe1-8587-c25fe1939350 None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:38.870073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908ca27d-a65c-4fe1-8587-c25fe1939350 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:38.871442 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908ca27d-a65c-4fe1-8587-c25fe1939350 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:38.878389 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 457/944] 10.222.0.102 () {56 vars in 1092 bytes} [Wed Aug 24 15:34:38 2022] GET /identity/v3/services => generated 1504 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:34:38.882828 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c46afbe-1218-4f17-8fe5-d6a85e3963a4 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:38.883120 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c46afbe-1218-4f17-8fe5-d6a85e3963a4 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:38.883398 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c46afbe-1218-4f17-8fe5-d6a85e3963a4 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:38.885115 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 488/945] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:38 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:38.891466 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b7753d-1f85-4549-81ff-cd70b88b8017 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:38.891805 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b7753d-1f85-4549-81ff-cd70b88b8017 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:38.892053 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b7753d-1f85-4549-81ff-cd70b88b8017 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:38.898730 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:38.915370 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-23b7753d-1f85-4549-81ff-cd70b88b8017 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:38.916652 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-23b7753d-1f85-4549-81ff-cd70b88b8017 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:38.996284 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 458/946] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:38 2022] POST /identity/v3/auth/tokens => generated 1476 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:34:39.285071 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c46afbe-1218-4f17-8fe5-d6a85e3963a4 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:39.324632 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:39.418379 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c46afbe-1218-4f17-8fe5-d6a85e3963a4 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:39.460608 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad81406-c977-4674-b212-76e739edd69f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:39.461473 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad81406-c977-4674-b212-76e739edd69f None admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:39.462003 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad81406-c977-4674-b212-76e739edd69f None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:39.462426 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad81406-c977-4674-b212-76e739edd69f None admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:39.463260 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad81406-c977-4674-b212-76e739edd69f None admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:39.464907 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad81406-c977-4674-b212-76e739edd69f None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:39.467735 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: WARNING keystone.server.flask.common [None req-fad81406-c977-4674-b212-76e739edd69f None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Aug 24 15:34:39.488108 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.sql.core [None req-fad81406-c977-4674-b212-76e739edd69f None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 24 15:34:39.488108 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Aug 24 15:34:39.488108 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [parameters: {'id': 'c89c46e002b242b1ba4abc456772101a', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Aug 24 15:34:39.488108 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=88217) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 24 15:34:39.509603 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-fad81406-c977-4674-b212-76e739edd69f None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id c89c46e002b242b1ba4abc456772101a.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id c89c46e002b242b1ba4abc456772101a. Aug 24 15:34:39.509603 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:39.509603 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 24 15:34:39.509603 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 24 15:34:39.509603 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 24 15:34:39.509603 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 24 15:34:39.509603 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 24 15:34:39.509603 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application result = self._query(query) Aug 24 15:34:39.509603 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 24 15:34:39.509603 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application conn.query(q) Aug 24 15:34:39.509603 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 24 15:34:39.509603 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 24 15:34:39.509603 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 24 15:34:39.510688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application result.read() Aug 24 15:34:39.510688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 24 15:34:39.510688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 24 15:34:39.510688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 24 15:34:39.510688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 24 15:34:39.510688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 24 15:34:39.510688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 24 15:34:39.510688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 24 15:34:39.510688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 24 15:34:39.510688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 24 15:34:39.510688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:39.510688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 24 15:34:39.510688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:39.510688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:39.510688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 24 15:34:39.510688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 24 15:34:39.510688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Aug 24 15:34:39.510688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return project_ref.to_dict() Aug 24 15:34:39.511862 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 24 15:34:39.511862 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application next(self.gen) Aug 24 15:34:39.511862 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 24 15:34:39.511862 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application yield resource Aug 24 15:34:39.511862 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 24 15:34:39.511862 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application next(self.gen) Aug 24 15:34:39.511862 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 24 15:34:39.511862 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self.session.rollback() Aug 24 15:34:39.511862 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 24 15:34:39.511862 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self.force_reraise() Aug 24 15:34:39.511862 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 24 15:34:39.511862 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise self.value Aug 24 15:34:39.511862 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 24 15:34:39.511862 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 24 15:34:39.511862 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 24 15:34:39.511862 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application session.commit() Aug 24 15:34:39.511862 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 24 15:34:39.512979 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 24 15:34:39.512979 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 24 15:34:39.512979 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self._prepare_impl() Aug 24 15:34:39.512979 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 24 15:34:39.512979 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self.session.flush() Aug 24 15:34:39.512979 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 24 15:34:39.512979 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self._flush(objects) Aug 24 15:34:39.512979 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 24 15:34:39.512979 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 24 15:34:39.512979 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 24 15:34:39.512979 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application compat.raise_( Aug 24 15:34:39.512979 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 24 15:34:39.512979 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception Aug 24 15:34:39.512979 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 24 15:34:39.512979 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application flush_context.execute() Aug 24 15:34:39.512979 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 24 15:34:39.512979 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rec.execute(self) Aug 24 15:34:39.515594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 24 15:34:39.515594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 24 15:34:39.515594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 24 15:34:39.515594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 24 15:34:39.515594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Aug 24 15:34:39.515594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application c = connection._execute_20( Aug 24 15:34:39.515594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 24 15:34:39.515594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 24 15:34:39.515594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 24 15:34:39.515594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 24 15:34:39.515594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 24 15:34:39.515594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 24 15:34:39.515594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 24 15:34:39.515594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 24 15:34:39.515594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 24 15:34:39.517043 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 24 15:34:39.517043 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 24 15:34:39.517043 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception Aug 24 15:34:39.517043 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 24 15:34:39.517043 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 24 15:34:39.517043 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 24 15:34:39.517043 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 24 15:34:39.517043 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 24 15:34:39.517043 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application result = self._query(query) Aug 24 15:34:39.517043 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 24 15:34:39.517043 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application conn.query(q) Aug 24 15:34:39.517043 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 24 15:34:39.517043 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 24 15:34:39.517043 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 24 15:34:39.517043 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application result.read() Aug 24 15:34:39.517043 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 24 15:34:39.517043 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 24 15:34:39.518024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 24 15:34:39.518024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 24 15:34:39.518024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 24 15:34:39.518024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 24 15:34:39.518024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 24 15:34:39.518024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 24 15:34:39.518024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 24 15:34:39.518024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Aug 24 15:34:39.518024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [parameters: {'id': 'c89c46e002b242b1ba4abc456772101a', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Aug 24 15:34:39.518024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 24 15:34:39.518024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:39.518024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 15:34:39.518024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:39.518024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:39.518024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Aug 24 15:34:39.518830 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Aug 24 15:34:39.518830 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 24 15:34:39.518830 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 24 15:34:39.518830 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Aug 24 15:34:39.518830 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:39.518830 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 15:34:39.518830 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:39.518830 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:39.518830 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:39.518830 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:39.518830 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:39.518830 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:39.518830 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:39.518830 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:39.518830 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:39.518830 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:39.518830 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:39.519755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:39.519755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Aug 24 15:34:39.519755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Aug 24 15:34:39.519755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:39.519755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:39.519755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Aug 24 15:34:39.519755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.Conflict( Aug 24 15:34:39.519755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id c89c46e002b242b1ba4abc456772101a. Aug 24 15:34:39.519755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:39.522170 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 489/947] 10.222.0.102 () {60 vars in 1145 bytes} [Wed Aug 24 15:34:39 2022] POST /identity/v3/projects => generated 262 bytes in 237 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:39.527762 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b7753d-1f85-4549-81ff-cd70b88b8017 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:39.637095 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b7753d-1f85-4549-81ff-cd70b88b8017 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:39.668705 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:39.675318 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380ffb6e-aea5-4309-9870-e988f7c3ece3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:39.675989 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380ffb6e-aea5-4309-9870-e988f7c3ece3 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:39.676269 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380ffb6e-aea5-4309-9870-e988f7c3ece3 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:39.676539 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380ffb6e-aea5-4309-9870-e988f7c3ece3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:39.684271 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-380ffb6e-aea5-4309-9870-e988f7c3ece3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:39.685733 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-380ffb6e-aea5-4309-9870-e988f7c3ece3 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:39.700118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-380ffb6e-aea5-4309-9870-e988f7c3ece3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:39.700118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:39.700118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:39.700118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:39.700118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:39.700118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:39.700118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:39.700118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:39.700118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:39.700118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:39.700118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:39.700118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:39.700118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:34:39.700118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:34:39.700118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:34:39.702418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:34:39.702418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:39.702418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:39.702418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:39.702418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:39.702418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:39.702418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:39.702418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:39.702418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:39.702418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:39.702418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:39.702418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:39.702418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:39.702418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:39.702418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:39.702418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:39.702418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:39.704313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:34:39.704313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:34:39.704313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:34:39.704313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:34:39.704313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:34:39.704313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:34:39.704313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:39.704313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:39.704313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 459/948] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:34:39 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:39.709881 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad81406-c977-4674-b212-76e739edd69f None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:39.838651 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:39.858054 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad81406-c977-4674-b212-76e739edd69f None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:39.897928 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c175c23-f5fc-404b-8bcc-be4eb106f836 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:39.898839 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c175c23-f5fc-404b-8bcc-be4eb106f836 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:39.899316 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c175c23-f5fc-404b-8bcc-be4eb106f836 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:39.899832 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c175c23-f5fc-404b-8bcc-be4eb106f836 None admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:39.900708 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c175c23-f5fc-404b-8bcc-be4eb106f836 None admin] RBAC: Adding query filter params (name=service) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:39.901195 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c175c23-f5fc-404b-8bcc-be4eb106f836 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:39.903022 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c175c23-f5fc-404b-8bcc-be4eb106f836 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:39.925641 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 490/949] 10.222.0.102 () {56 vars in 1117 bytes} [Wed Aug 24 15:34:39 2022] GET /identity/v3/projects?name=service => generated 413 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:40.306976 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d65ea48-105b-4944-aa23-3f225de87e36 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:40.307173 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d65ea48-105b-4944-aa23-3f225de87e36 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:40.307607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d65ea48-105b-4944-aa23-3f225de87e36 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:40.309333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 460/950] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:40 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:40.316329 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1746e08a-09ae-4557-9f4e-4984fb28b38e None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:40.316856 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1746e08a-09ae-4557-9f4e-4984fb28b38e None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:40.316856 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1746e08a-09ae-4557-9f4e-4984fb28b38e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:40.336717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-1746e08a-09ae-4557-9f4e-4984fb28b38e None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:40.338597 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-1746e08a-09ae-4557-9f4e-4984fb28b38e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:40.416708 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 491/951] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:40 2022] POST /identity/v3/auth/tokens => generated 1476 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:34:40.688587 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d65ea48-105b-4944-aa23-3f225de87e36 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:40.728952 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b358135c-f94c-48fb-9a43-5d83fd5206d5 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:40.729653 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b358135c-f94c-48fb-9a43-5d83fd5206d5 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:40.730159 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b358135c-f94c-48fb-9a43-5d83fd5206d5 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:40.732775 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 492/952] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:40 2022] GET /identity => generated 271 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:34:40.739675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbcb646-e40d-47f8-bdc3-5a96fb894187 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:40.739675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbcb646-e40d-47f8-bdc3-5a96fb894187 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:40.739675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbcb646-e40d-47f8-bdc3-5a96fb894187 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:40.759436 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-4cbcb646-e40d-47f8-bdc3-5a96fb894187 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:40.760883 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-4cbcb646-e40d-47f8-bdc3-5a96fb894187 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:40.797252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d65ea48-105b-4944-aa23-3f225de87e36 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:40.834692 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2a40ab-64ef-46dc-9080-c8aa0a86a023 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:40.835332 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2a40ab-64ef-46dc-9080-c8aa0a86a023 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:40.835611 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2a40ab-64ef-46dc-9080-c8aa0a86a023 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:40.835911 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2a40ab-64ef-46dc-9080-c8aa0a86a023 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:40.847735 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 493/953] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:40 2022] POST /identity/v3/auth/tokens => generated 1476 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 15:34:40.856289 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2a40ab-64ef-46dc-9080-c8aa0a86a023 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:34:40.862060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-9a2a40ab-64ef-46dc-9080-c8aa0a86a023 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 24 15:34:40.862060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:40.862060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:40.862060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:40.862060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:40.862060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:40.862060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:40.862060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:40.862060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:40.862060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:40.862060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:40.862060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:40.862060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:34:40.862060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:34:40.862060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:34:40.862060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:34:40.862989 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:34:40.862989 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:34:40.862989 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:34:40.862989 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:34:40.862989 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:40.862989 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:40.862989 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:40.862989 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:40.862989 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:40.862989 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:40.862989 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:40.862989 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:40.862989 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:40.862989 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:40.862989 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:40.862989 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:40.864036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:40.864036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:40.864036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:40.864036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:40.864036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:34:40.864036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:34:40.864036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:34:40.864036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:34:40.864036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:34:40.864036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:34:40.864036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 24 15:34:40.864036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:40.865249 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 461/954] 10.222.0.102 () {56 vars in 1125 bytes} [Wed Aug 24 15:34:40 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:40.875684 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbcb646-e40d-47f8-bdc3-5a96fb894187 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:40.885089 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:40.949124 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e26d73-69dc-4258-b78e-86a32928a6ef None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:40.949517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e26d73-69dc-4258-b78e-86a32928a6ef None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:40.949892 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e26d73-69dc-4258-b78e-86a32928a6ef None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:40.955291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 462/955] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:40 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:34:40.958151 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df751a03-8cdc-4b4a-a4f2-3911a0f992d8 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:40.958546 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df751a03-8cdc-4b4a-a4f2-3911a0f992d8 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:40.958935 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df751a03-8cdc-4b4a-a4f2-3911a0f992d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:40.978001 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-df751a03-8cdc-4b4a-a4f2-3911a0f992d8 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:40.979154 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-df751a03-8cdc-4b4a-a4f2-3911a0f992d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:40.992091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbcb646-e40d-47f8-bdc3-5a96fb894187 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:41.029789 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3e06ed-68f5-4922-a206-682046d7be81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:41.031317 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3e06ed-68f5-4922-a206-682046d7be81 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:41.031777 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3e06ed-68f5-4922-a206-682046d7be81 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:41.032191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3e06ed-68f5-4922-a206-682046d7be81 None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:41.033040 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a3e06ed-68f5-4922-a206-682046d7be81 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:41.033491 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a3e06ed-68f5-4922-a206-682046d7be81 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:41.035027 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a3e06ed-68f5-4922-a206-682046d7be81 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:41.035345 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:41.045486 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 494/956] 10.222.0.102 () {56 vars in 1120 bytes} [Wed Aug 24 15:34:40 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 340 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:41.051354 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3e06ed-68f5-4922-a206-682046d7be81 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:41.065388 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 463/957] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:40 2022] POST /identity/v3/auth/tokens => generated 1476 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 15:34:41.122347 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df751a03-8cdc-4b4a-a4f2-3911a0f992d8 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:41.131427 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:41.181716 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3e06ed-68f5-4922-a206-682046d7be81 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:41.224974 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a5c843-7909-4c4e-9495-c33780c3edba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:41.226155 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a5c843-7909-4c4e-9495-c33780c3edba None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:41.226642 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a5c843-7909-4c4e-9495-c33780c3edba None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:41.227152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a5c843-7909-4c4e-9495-c33780c3edba None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:41.239942 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a5c843-7909-4c4e-9495-c33780c3edba None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:41.241853 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a5c843-7909-4c4e-9495-c33780c3edba None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:41.248157 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df751a03-8cdc-4b4a-a4f2-3911a0f992d8 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:41.259418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-34a5c843-7909-4c4e-9495-c33780c3edba None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:41.259418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:41.259418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:41.259418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:41.259418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:41.259418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:41.259418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:41.259418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:41.259418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:41.259418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:41.259418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:41.259418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:41.259418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:41.259418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:41.259418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:41.259418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:41.260471 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:41.260471 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:41.260471 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:41.260471 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:41.260471 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:41.260471 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:41.260471 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:41.260471 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:41.260471 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:41.260471 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:41.260471 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:41.260471 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:41.260471 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:41.260471 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:41.260471 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:41.260471 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:41.260471 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:41.261369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:41.261369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:41.261369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:41.263689 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 495/958] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:34:41.270148 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a5c843-7909-4c4e-9495-c33780c3edba None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:41.299360 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1328c6-a342-4801-ad03-5e3ef740f08c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:41.300059 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1328c6-a342-4801-ad03-5e3ef740f08c None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:41.300342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1328c6-a342-4801-ad03-5e3ef740f08c None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:41.300621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1328c6-a342-4801-ad03-5e3ef740f08c None admin] PATH_INFO: `/v3/services/compute` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:41.301627 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1328c6-a342-4801-ad03-5e3ef740f08c None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:41.303004 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1328c6-a342-4801-ad03-5e3ef740f08c None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:41.316111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-7b1328c6-a342-4801-ad03-5e3ef740f08c None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 24 15:34:41.316111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:41.316111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 15:34:41.316111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 15:34:41.316111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 15:34:41.316111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 15:34:41.316111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 15:34:41.316111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:41.316111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 24 15:34:41.316111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:41.316111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 15:34:41.316111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:41.316111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:41.316111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:41.316111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:41.316111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:41.316111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:41.317350 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:41.317350 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:41.317350 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:41.317350 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:41.317350 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:41.317350 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:41.317350 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 15:34:41.317350 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 15:34:41.317350 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 15:34:41.317350 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 15:34:41.317350 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:41.317350 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:41.317350 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:41.317350 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:41.317350 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:41.317350 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:41.318212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:41.318212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:41.318212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:41.318212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:41.318212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:41.318212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:41.318212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:41.318212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:41.318212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:41.318212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:41.318212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 15:34:41.318212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:41.318212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 24 15:34:41.318212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:41.322555 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 464/959] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:34:41 2022] GET /identity/v3/services/compute => generated 88 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:34:41.323903 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1328c6-a342-4801-ad03-5e3ef740f08c None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:41.387343 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:41.409069 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a5c843-7909-4c4e-9495-c33780c3edba None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:41.432228 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:41.451990 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4836092c-daab-42f7-ba3f-8c3cc92d986d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:41.453447 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4836092c-daab-42f7-ba3f-8c3cc92d986d None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:41.453730 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4836092c-daab-42f7-ba3f-8c3cc92d986d None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:41.454005 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4836092c-daab-42f7-ba3f-8c3cc92d986d None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:41.454683 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4836092c-daab-42f7-ba3f-8c3cc92d986d None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:41.454987 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4836092c-daab-42f7-ba3f-8c3cc92d986d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:41.456026 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1328c6-a342-4801-ad03-5e3ef740f08c None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:41.456380 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4836092c-daab-42f7-ba3f-8c3cc92d986d None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:41.470907 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 496/960] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:41 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:41.472274 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4836092c-daab-42f7-ba3f-8c3cc92d986d None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:41.503184 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82f6691-e1f9-4fb3-a127-8e8f96eaa57e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:41.504112 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82f6691-e1f9-4fb3-a127-8e8f96eaa57e None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:41.504566 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82f6691-e1f9-4fb3-a127-8e8f96eaa57e None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:41.505016 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82f6691-e1f9-4fb3-a127-8e8f96eaa57e None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:41.505651 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a82f6691-e1f9-4fb3-a127-8e8f96eaa57e None admin] RBAC: Adding query filter params (name=compute) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:41.505972 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a82f6691-e1f9-4fb3-a127-8e8f96eaa57e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:41.507311 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a82f6691-e1f9-4fb3-a127-8e8f96eaa57e None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:41.512473 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 465/961] 10.222.0.102 () {56 vars in 1117 bytes} [Wed Aug 24 15:34:41 2022] GET /identity/v3/services?name=compute => generated 125 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:41.513292 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82f6691-e1f9-4fb3-a127-8e8f96eaa57e None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:41.586282 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4836092c-daab-42f7-ba3f-8c3cc92d986d None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:41.623993 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d48eed-868f-4d74-a509-e1af4e6bb62d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:41.624897 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d48eed-868f-4d74-a509-e1af4e6bb62d None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:41.625139 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d48eed-868f-4d74-a509-e1af4e6bb62d None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:41.625411 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d48eed-868f-4d74-a509-e1af4e6bb62d None admin] PATH_INFO: `/v3/services/object-store` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:41.626000 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d48eed-868f-4d74-a509-e1af4e6bb62d None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:41.627119 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82f6691-e1f9-4fb3-a127-8e8f96eaa57e None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:41.627368 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d48eed-868f-4d74-a509-e1af4e6bb62d None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:41.636351 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-d7d48eed-868f-4d74-a509-e1af4e6bb62d None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 24 15:34:41.636351 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:41.636351 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 15:34:41.636351 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 15:34:41.636351 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 15:34:41.636351 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 15:34:41.636351 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 15:34:41.636351 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:41.636351 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 24 15:34:41.636351 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:41.636351 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 15:34:41.636351 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:41.636351 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:41.636351 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:41.636351 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:41.636351 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:41.636351 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:41.637400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:41.637400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:41.637400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:41.637400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:41.637400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:41.637400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:41.637400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 15:34:41.637400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 15:34:41.637400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 15:34:41.637400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 15:34:41.637400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:41.637400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:41.637400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:41.637400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:41.637400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:41.637400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:41.638249 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:41.638249 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:41.638249 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:41.638249 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:41.638249 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:41.638249 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:41.638249 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:41.638249 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:41.638249 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:41.638249 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:41.638249 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 15:34:41.638249 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:41.638249 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 24 15:34:41.638249 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:41.642407 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:41.643214 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 497/962] 10.222.0.102 () {56 vars in 1131 bytes} [Wed Aug 24 15:34:41 2022] GET /identity/v3/services/object-store => generated 93 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:41.644601 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d48eed-868f-4d74-a509-e1af4e6bb62d None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:41.676911 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda82a62-062d-47ba-8a65-745ea474359b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:41.677564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda82a62-062d-47ba-8a65-745ea474359b None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:41.677868 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda82a62-062d-47ba-8a65-745ea474359b None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:41.678116 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda82a62-062d-47ba-8a65-745ea474359b None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:41.686175 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda82a62-062d-47ba-8a65-745ea474359b None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:41.687832 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda82a62-062d-47ba-8a65-745ea474359b None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:41.696012 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:41.700045 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-bda82a62-062d-47ba-8a65-745ea474359b None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 24 15:34:41.700045 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:41.700045 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:41.700045 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:41.700045 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:41.700045 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:41.700045 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:41.700045 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:41.700045 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:41.700045 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:41.700045 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:41.700045 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:41.700045 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:34:41.700045 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:34:41.700045 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:34:41.700045 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:34:41.701281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:41.701281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:41.701281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:34:41.701281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:41.701281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:34:41.701281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:41.701281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:41.701281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:41.701281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:41.701281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:41.701281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:41.701281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:41.701281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:41.701281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:41.701281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:41.701281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:41.701281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:41.701281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:41.702313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:41.702313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:41.702313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:34:41.702313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:34:41.702313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:34:41.702313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:34:41.702313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:34:41.702313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:34:41.702313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 24 15:34:41.702313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:41.702313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 466/963] 10.222.0.102 () {56 vars in 1122 bytes} [Wed Aug 24 15:34:41 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:41.702313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda82a62-062d-47ba-8a65-745ea474359b None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:41.761797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d48eed-868f-4d74-a509-e1af4e6bb62d None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:41.800523 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9219ef7-116b-4c4c-b804-ad01908df2eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:41.801238 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9219ef7-116b-4c4c-b804-ad01908df2eb None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:41.801518 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9219ef7-116b-4c4c-b804-ad01908df2eb None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:41.801790 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9219ef7-116b-4c4c-b804-ad01908df2eb None admin] PATH_INFO: `/v3/services` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:41.802442 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9219ef7-116b-4c4c-b804-ad01908df2eb None admin] RBAC: Adding query filter params (type=compute) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:41.802743 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9219ef7-116b-4c4c-b804-ad01908df2eb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:41.804211 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9219ef7-116b-4c4c-b804-ad01908df2eb None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:41.820029 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 498/964] 10.222.0.102 () {56 vars in 1117 bytes} [Wed Aug 24 15:34:41 2022] GET /identity/v3/services?type=compute => generated 125 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:41.820029 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9219ef7-116b-4c4c-b804-ad01908df2eb None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:41.829446 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda82a62-062d-47ba-8a65-745ea474359b None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:41.876088 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cf72f3-4e6b-47a1-8923-e8221687fdd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:41.876980 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cf72f3-4e6b-47a1-8923-e8221687fdd5 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:41.877415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cf72f3-4e6b-47a1-8923-e8221687fdd5 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:41.877837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cf72f3-4e6b-47a1-8923-e8221687fdd5 None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:41.878672 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cf72f3-4e6b-47a1-8923-e8221687fdd5 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:41.879120 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cf72f3-4e6b-47a1-8923-e8221687fdd5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:41.880822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cf72f3-4e6b-47a1-8923-e8221687fdd5 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:41.894220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 467/965] 10.222.0.102 () {56 vars in 1127 bytes} [Wed Aug 24 15:34:41 2022] GET /identity/v3/services?name=object-store => generated 130 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:41.896174 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cf72f3-4e6b-47a1-8923-e8221687fdd5 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:41.935739 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:41.958813 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9219ef7-116b-4c4c-b804-ad01908df2eb None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:41.973393 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:42.010706 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6af3012-0499-4420-bc05-eaad97643e7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:42.011333 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6af3012-0499-4420-bc05-eaad97643e7c None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:42.011664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6af3012-0499-4420-bc05-eaad97643e7c None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:42.012066 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6af3012-0499-4420-bc05-eaad97643e7c None admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:42.012675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6af3012-0499-4420-bc05-eaad97643e7c None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:42.012983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6af3012-0499-4420-bc05-eaad97643e7c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:42.014483 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6af3012-0499-4420-bc05-eaad97643e7c None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:42.017895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cf72f3-4e6b-47a1-8923-e8221687fdd5 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:42.036686 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 499/966] 10.222.0.102 () {56 vars in 1154 bytes} [Wed Aug 24 15:34:41 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 386 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:42.037992 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e138ceec-bb8a-495f-b5b5-8cc4c1e59272 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:42.038428 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e138ceec-bb8a-495f-b5b5-8cc4c1e59272 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:42.038606 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e138ceec-bb8a-495f-b5b5-8cc4c1e59272 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:42.040009 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 500/967] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:42 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:42.044781 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e138ceec-bb8a-495f-b5b5-8cc4c1e59272 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:42.060917 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85558648-6d18-472e-9890-9a3854f64b6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:42.061530 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85558648-6d18-472e-9890-9a3854f64b6c None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:42.061807 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85558648-6d18-472e-9890-9a3854f64b6c None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:42.062750 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85558648-6d18-472e-9890-9a3854f64b6c None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:42.063445 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85558648-6d18-472e-9890-9a3854f64b6c None admin] RBAC: Adding query filter params (type=object-store) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:42.063783 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85558648-6d18-472e-9890-9a3854f64b6c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:42.065149 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85558648-6d18-472e-9890-9a3854f64b6c None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:42.071050 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 468/968] 10.222.0.102 () {56 vars in 1127 bytes} [Wed Aug 24 15:34:41 2022] GET /identity/v3/services?type=object-store => generated 358 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:42.073630 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f05e271-df38-487c-97e2-c4e555ab8178 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:42.074118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f05e271-df38-487c-97e2-c4e555ab8178 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:42.074716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f05e271-df38-487c-97e2-c4e555ab8178 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:42.095074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-3f05e271-df38-487c-97e2-c4e555ab8178 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:42.096322 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-3f05e271-df38-487c-97e2-c4e555ab8178 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:42.157108 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e138ceec-bb8a-495f-b5b5-8cc4c1e59272 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:42.176227 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 469/969] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:42 2022] POST /identity/v3/auth/tokens => generated 1476 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:34:42.177472 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f05e271-df38-487c-97e2-c4e555ab8178 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:42.196135 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a58806-4b95-40c9-9d7d-3c130b1c904c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:42.197001 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a58806-4b95-40c9-9d7d-3c130b1c904c None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:42.197505 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a58806-4b95-40c9-9d7d-3c130b1c904c None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:42.198032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a58806-4b95-40c9-9d7d-3c130b1c904c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:42.222754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:42.240418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:42.285751 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a58806-4b95-40c9-9d7d-3c130b1c904c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:42.287715 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a58806-4b95-40c9-9d7d-3c130b1c904c None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:42.318498 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f05e271-df38-487c-97e2-c4e555ab8178 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:42.375735 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc110a30-a975-44c2-bfb0-28add9e9967e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:42.376800 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc110a30-a975-44c2-bfb0-28add9e9967e None admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:42.377336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc110a30-a975-44c2-bfb0-28add9e9967e None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:42.377688 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc110a30-a975-44c2-bfb0-28add9e9967e None admin] PATH_INFO: `/v3/endpoints` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:42.378379 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc110a30-a975-44c2-bfb0-28add9e9967e None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:42.379790 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc110a30-a975-44c2-bfb0-28add9e9967e None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:42.411122 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 470/970] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:34:42 2022] POST /identity/v3/endpoints => generated 351 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:34:42.425011 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 501/971] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:34:42 2022] GET /identity/v3/auth/tokens => generated 1476 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 15:34:42.430970 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc110a30-a975-44c2-bfb0-28add9e9967e None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:42.471025 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a58806-4b95-40c9-9d7d-3c130b1c904c None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:42.543976 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:42.562976 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:42.579066 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc110a30-a975-44c2-bfb0-28add9e9967e None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:42.614942 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a58806-4b95-40c9-9d7d-3c130b1c904c None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:42.627012 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb900829-5df3-4311-bc48-b8486aad0f98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:42.627723 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb900829-5df3-4311-bc48-b8486aad0f98 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:42.628131 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb900829-5df3-4311-bc48-b8486aad0f98 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:42.628404 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb900829-5df3-4311-bc48-b8486aad0f98 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:42.639709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb900829-5df3-4311-bc48-b8486aad0f98 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:42.642040 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb900829-5df3-4311-bc48-b8486aad0f98 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:42.653547 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d608b6-aff3-47aa-b80d-7288cf1656ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:42.654470 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d608b6-aff3-47aa-b80d-7288cf1656ae None admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:42.654879 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d608b6-aff3-47aa-b80d-7288cf1656ae None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:42.655169 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d608b6-aff3-47aa-b80d-7288cf1656ae None admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:42.656457 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d608b6-aff3-47aa-b80d-7288cf1656ae None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:42.657984 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d608b6-aff3-47aa-b80d-7288cf1656ae None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:42.659762 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-eb900829-5df3-4311-bc48-b8486aad0f98 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:42.659762 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:42.659762 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:42.659762 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:42.659762 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:42.659762 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:42.659762 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:42.659762 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:42.659762 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:42.659762 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:42.659762 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:42.659762 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:42.659762 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:42.659762 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:42.659762 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:42.659762 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:42.660714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:42.660714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:42.660714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:42.660714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:42.660714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:42.660714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:42.660714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:42.660714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:42.660714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:42.660714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:42.660714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:42.660714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:42.660714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:42.660714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:42.660714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:42.660714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:42.660714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:42.661679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:42.661679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:42.661679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:42.661679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 471/972] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:42.666466 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb900829-5df3-4311-bc48-b8486aad0f98 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:42.679451 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.sql.core [None req-38d608b6-aff3-47aa-b80d-7288cf1656ae None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 24 15:34:42.679451 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Aug 24 15:34:42.679451 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [parameters: {'user_id': '1b8378bfb4e84af6addbbb07e5143749', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Aug 24 15:34:42.679451 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=88217) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 24 15:34:42.687568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-38d608b6-aff3-47aa-b80d-7288cf1656ae None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Aug 24 15:34:42.687568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:42.687568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 24 15:34:42.687568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 24 15:34:42.687568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 24 15:34:42.687568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 24 15:34:42.687568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 24 15:34:42.687568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application result = self._query(query) Aug 24 15:34:42.687568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 24 15:34:42.687568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application conn.query(q) Aug 24 15:34:42.687568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 24 15:34:42.687568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 24 15:34:42.687568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 24 15:34:42.687568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application result.read() Aug 24 15:34:42.687568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 24 15:34:42.689208 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 24 15:34:42.689208 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 24 15:34:42.689208 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 24 15:34:42.689208 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 24 15:34:42.689208 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 24 15:34:42.689208 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 24 15:34:42.689208 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 24 15:34:42.689208 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 24 15:34:42.689208 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:42.689208 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 24 15:34:42.689208 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:42.689208 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:42.689208 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 24 15:34:42.689208 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 24 15:34:42.689208 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Aug 24 15:34:42.689208 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Aug 24 15:34:42.689208 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 24 15:34:42.689208 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application next(self.gen) Aug 24 15:34:42.690182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 24 15:34:42.690182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application yield resource Aug 24 15:34:42.690182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 24 15:34:42.690182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application next(self.gen) Aug 24 15:34:42.690182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 24 15:34:42.690182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self.session.rollback() Aug 24 15:34:42.690182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 24 15:34:42.690182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self.force_reraise() Aug 24 15:34:42.690182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 24 15:34:42.690182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise self.value Aug 24 15:34:42.690182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 24 15:34:42.690182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 24 15:34:42.690182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 24 15:34:42.690182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application session.commit() Aug 24 15:34:42.690182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 24 15:34:42.690182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 24 15:34:42.690182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 24 15:34:42.691872 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self._prepare_impl() Aug 24 15:34:42.691872 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 24 15:34:42.691872 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self.session.flush() Aug 24 15:34:42.691872 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 24 15:34:42.691872 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self._flush(objects) Aug 24 15:34:42.691872 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 24 15:34:42.691872 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 24 15:34:42.691872 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 24 15:34:42.691872 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application compat.raise_( Aug 24 15:34:42.691872 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 24 15:34:42.691872 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception Aug 24 15:34:42.691872 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 24 15:34:42.691872 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application flush_context.execute() Aug 24 15:34:42.691872 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 24 15:34:42.691872 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rec.execute(self) Aug 24 15:34:42.691872 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 24 15:34:42.691872 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 24 15:34:42.693146 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 24 15:34:42.693146 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 24 15:34:42.693146 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Aug 24 15:34:42.693146 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application result = connection._execute_20( Aug 24 15:34:42.693146 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 24 15:34:42.693146 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 24 15:34:42.693146 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 24 15:34:42.693146 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 24 15:34:42.693146 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 24 15:34:42.693146 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 24 15:34:42.693146 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 24 15:34:42.693146 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 24 15:34:42.693146 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 24 15:34:42.693146 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 24 15:34:42.693146 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 24 15:34:42.693146 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception Aug 24 15:34:42.694253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 24 15:34:42.694253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 24 15:34:42.694253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 24 15:34:42.694253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 24 15:34:42.694253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 24 15:34:42.694253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application result = self._query(query) Aug 24 15:34:42.694253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 24 15:34:42.694253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application conn.query(q) Aug 24 15:34:42.694253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 24 15:34:42.694253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 24 15:34:42.694253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 24 15:34:42.694253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application result.read() Aug 24 15:34:42.694253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 24 15:34:42.694253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 24 15:34:42.694253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 24 15:34:42.694253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 24 15:34:42.695400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 24 15:34:42.695400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 24 15:34:42.695400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 24 15:34:42.695400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 24 15:34:42.695400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 24 15:34:42.695400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Aug 24 15:34:42.695400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [parameters: {'user_id': '1b8378bfb4e84af6addbbb07e5143749', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Aug 24 15:34:42.695400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 24 15:34:42.695400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:42.695400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 15:34:42.695400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:42.695400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:42.695400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:42.695400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:42.695400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:42.696413 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:42.696413 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:42.696413 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:42.696413 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:42.696413 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:42.696413 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:42.696413 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:42.696413 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Aug 24 15:34:42.696413 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Aug 24 15:34:42.696413 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:42.696413 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:42.696413 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:34:42.696413 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:42.696413 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:34:42.696413 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:42.696413 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Aug 24 15:34:42.696413 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Aug 24 15:34:42.697569 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Aug 24 15:34:42.697569 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Aug 24 15:34:42.697569 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 24 15:34:42.697569 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 24 15:34:42.697569 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Aug 24 15:34:42.697569 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:42.697569 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 502/973] 10.222.0.102 () {60 vars in 1136 bytes} [Wed Aug 24 15:34:42 2022] POST /identity/v3/users => generated 160 bytes in 219 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:34:42.706864 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d608b6-aff3-47aa-b80d-7288cf1656ae None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:42.799701 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb900829-5df3-4311-bc48-b8486aad0f98 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:42.824548 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d608b6-aff3-47aa-b80d-7288cf1656ae None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:42.844196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:42.852680 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e036c5a-a73f-4faa-9725-83410c61b998 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:42.853501 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:42.853799 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e036c5a-a73f-4faa-9725-83410c61b998 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:42.854288 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e036c5a-a73f-4faa-9725-83410c61b998 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:42.854755 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e036c5a-a73f-4faa-9725-83410c61b998 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:42.856357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e036c5a-a73f-4faa-9725-83410c61b998 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:42.857071 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e036c5a-a73f-4faa-9725-83410c61b998 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:42.860731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e036c5a-a73f-4faa-9725-83410c61b998 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:42.866105 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebb76ad-87ce-40a4-bb3b-f45b9252dced None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:42.866776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebb76ad-87ce-40a4-bb3b-f45b9252dced None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:42.867059 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebb76ad-87ce-40a4-bb3b-f45b9252dced None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:42.867373 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebb76ad-87ce-40a4-bb3b-f45b9252dced None admin] PATH_INFO: `/v3/users/cinder` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:42.877749 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ebb76ad-87ce-40a4-bb3b-f45b9252dced None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:42.877994 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 472/974] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:42 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:42.883862 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ebb76ad-87ce-40a4-bb3b-f45b9252dced None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:42.884048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e036c5a-a73f-4faa-9725-83410c61b998 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:42.897595 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-9ebb76ad-87ce-40a4-bb3b-f45b9252dced None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 24 15:34:42.897595 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:42.897595 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:42.897595 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:42.897595 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:42.897595 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:42.897595 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:42.897595 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:42.897595 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:42.897595 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:42.897595 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:42.897595 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:42.897595 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:34:42.897595 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:34:42.897595 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:34:42.897595 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:34:42.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:42.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:42.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:34:42.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:42.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:34:42.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:42.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:42.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:42.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:42.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:42.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:42.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:42.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:42.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:42.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:42.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:42.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:42.898780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:42.899932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:42.899932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:42.899932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:34:42.899932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:34:42.899932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:34:42.899932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:34:42.899932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:34:42.899932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:34:42.899932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 24 15:34:42.899932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:42.902238 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 503/975] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:34:42 2022] GET /identity/v3/users/cinder => generated 84 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:34:42.907811 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebb76ad-87ce-40a4-bb3b-f45b9252dced None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:43.019947 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e036c5a-a73f-4faa-9725-83410c61b998 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:43.021192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebb76ad-87ce-40a4-bb3b-f45b9252dced None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:43.060657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd958c4-3501-4759-a2dc-32019a943dbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:43.060866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335a2f94-76a9-4b85-9e4a-033ba274ac93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:43.061596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335a2f94-76a9-4b85-9e4a-033ba274ac93 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:43.061877 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd958c4-3501-4759-a2dc-32019a943dbe None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:43.062565 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335a2f94-76a9-4b85-9e4a-033ba274ac93 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:43.062859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335a2f94-76a9-4b85-9e4a-033ba274ac93 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:43.063579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd958c4-3501-4759-a2dc-32019a943dbe None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:43.064084 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd958c4-3501-4759-a2dc-32019a943dbe None admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:43.065073 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd958c4-3501-4759-a2dc-32019a943dbe None admin] RBAC: Adding query filter params (name=cinder) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:43.065531 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd958c4-3501-4759-a2dc-32019a943dbe None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:43.067498 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd958c4-3501-4759-a2dc-32019a943dbe None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:43.071070 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335a2f94-76a9-4b85-9e4a-033ba274ac93 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:43.072642 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335a2f94-76a9-4b85-9e4a-033ba274ac93 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:43.085173 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-335a2f94-76a9-4b85-9e4a-033ba274ac93 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:43.085173 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:43.085173 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:43.085173 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:43.085173 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:43.085173 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:43.085173 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:43.085173 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:43.085173 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:43.085173 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:43.085173 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:43.085173 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:43.085173 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:34:43.085173 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:34:43.085173 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:34:43.086330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:34:43.086330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:43.086330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:43.086330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:43.086330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:43.086330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:43.086330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:43.086330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:43.086330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:43.086330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:43.086330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:43.086330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:43.086330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:43.086330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:43.086330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:43.086330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:43.086330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:43.087331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:34:43.087331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:34:43.087331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:34:43.087331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:34:43.087331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:34:43.087331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:34:43.087331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:43.087331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:43.092610 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 473/976] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:34:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:43.094316 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335a2f94-76a9-4b85-9e4a-033ba274ac93 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:43.114852 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 504/977] 10.222.0.102 () {56 vars in 1106 bytes} [Wed Aug 24 15:34:42 2022] GET /identity/v3/users?name=cinder => generated 356 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:43.164247 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:43.228139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335a2f94-76a9-4b85-9e4a-033ba274ac93 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:43.270023 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd4c6eb-cc73-47a9-881d-6a062bca6624 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:43.270646 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd4c6eb-cc73-47a9-881d-6a062bca6624 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:43.270929 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd4c6eb-cc73-47a9-881d-6a062bca6624 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:43.271203 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd4c6eb-cc73-47a9-881d-6a062bca6624 None admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:43.271966 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cd4c6eb-cc73-47a9-881d-6a062bca6624 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:43.272343 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cd4c6eb-cc73-47a9-881d-6a062bca6624 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:43.273847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cd4c6eb-cc73-47a9-881d-6a062bca6624 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:43.291167 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 474/978] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:34:43 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:43.297662 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd958c4-3501-4759-a2dc-32019a943dbe None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:43.355949 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:43.419935 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd958c4-3501-4759-a2dc-32019a943dbe None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:43.459046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba48f3b-a289-4870-899e-4f1624ffa443 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:43.459771 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba48f3b-a289-4870-899e-4f1624ffa443 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:43.459999 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba48f3b-a289-4870-899e-4f1624ffa443 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:43.460233 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba48f3b-a289-4870-899e-4f1624ffa443 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:43.460975 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba48f3b-a289-4870-899e-4f1624ffa443 None admin] RBAC: Adding query filter params (role.id=5cd803f4cb3d42f0b728362ffd2f43cd, scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5, user.id=a6c17620ead14cb49d962990e1b1e0d8) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:43.461290 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba48f3b-a289-4870-899e-4f1624ffa443 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:43.462758 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba48f3b-a289-4870-899e-4f1624ffa443 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:43.469806 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 505/979] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:34:43 2022] GET /identity/v3/role_assignments?user.id=a6c17620ead14cb49d962990e1b1e0d8&scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5&role.id=5cd803f4cb3d42f0b728362ffd2f43cd => generated 610 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:43.826858 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee2ee6d-0100-40bc-8e73-d8b6cc098f92 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:43.827370 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee2ee6d-0100-40bc-8e73-d8b6cc098f92 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:43.827370 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee2ee6d-0100-40bc-8e73-d8b6cc098f92 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:43.829551 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 475/980] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:43 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:43.834979 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2c2044-d660-4b70-b6b8-43b8378b920a None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:43.835269 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2c2044-d660-4b70-b6b8-43b8378b920a None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:43.835547 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2c2044-d660-4b70-b6b8-43b8378b920a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:43.861279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-5a2c2044-d660-4b70-b6b8-43b8378b920a None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:43.862801 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-5a2c2044-d660-4b70-b6b8-43b8378b920a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:43.941654 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 506/981] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:43 2022] POST /identity/v3/auth/tokens => generated 1476 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:34:44.226534 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee2ee6d-0100-40bc-8e73-d8b6cc098f92 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:44.348280 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee2ee6d-0100-40bc-8e73-d8b6cc098f92 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:44.362230 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:44.386106 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2d1984-b5ad-4a3e-8a5e-4ca83aeb93e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:44.386708 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2d1984-b5ad-4a3e-8a5e-4ca83aeb93e0 None admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:44.386988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2d1984-b5ad-4a3e-8a5e-4ca83aeb93e0 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:44.387258 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2d1984-b5ad-4a3e-8a5e-4ca83aeb93e0 None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:44.387950 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2d1984-b5ad-4a3e-8a5e-4ca83aeb93e0 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:44.389328 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2d1984-b5ad-4a3e-8a5e-4ca83aeb93e0 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:44.406661 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 476/982] 10.222.0.102 () {60 vars in 1146 bytes} [Wed Aug 24 15:34:44 2022] POST /identity/v3/services => generated 243 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:34:44.560717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7195490b-521d-4c29-9422-00ebc99eabc0 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:44.561203 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7195490b-521d-4c29-9422-00ebc99eabc0 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:44.561203 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7195490b-521d-4c29-9422-00ebc99eabc0 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:44.562606 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 507/983] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:44 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:44.568761 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d3a44a-3085-45a0-b6bd-6bb1be7f49b4 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:44.569050 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d3a44a-3085-45a0-b6bd-6bb1be7f49b4 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:44.569353 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d3a44a-3085-45a0-b6bd-6bb1be7f49b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:44.588343 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-90d3a44a-3085-45a0-b6bd-6bb1be7f49b4 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:44.589818 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-90d3a44a-3085-45a0-b6bd-6bb1be7f49b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:44.673706 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 477/984] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:44 2022] POST /identity/v3/auth/tokens => generated 1572 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:34:44.970923 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7195490b-521d-4c29-9422-00ebc99eabc0 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:45.019018 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:45.083670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7195490b-521d-4c29-9422-00ebc99eabc0 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:45.120292 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a720a6-e385-4f81-af3e-185d55454353 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:45.120843 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a720a6-e385-4f81-af3e-185d55454353 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:45.121124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a720a6-e385-4f81-af3e-185d55454353 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:45.121397 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a720a6-e385-4f81-af3e-185d55454353 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:45.121997 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a720a6-e385-4f81-af3e-185d55454353 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:45.123368 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a720a6-e385-4f81-af3e-185d55454353 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:45.133024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-11a720a6-e385-4f81-af3e-185d55454353 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 24 15:34:45.133024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:45.133024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 15:34:45.133024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 15:34:45.133024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 15:34:45.133024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 15:34:45.133024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 15:34:45.133024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:45.133024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 24 15:34:45.133024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:45.133024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 15:34:45.133024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:45.133024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:45.133024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:45.133024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:45.133024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:45.133024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:45.134040 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:45.134040 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:45.134040 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:45.134040 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:45.134040 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:45.134040 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:45.134040 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 15:34:45.134040 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 15:34:45.134040 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 15:34:45.134040 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 15:34:45.134040 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:45.134040 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:45.134040 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:45.134040 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:45.134040 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:45.134040 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:45.136649 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:45.136649 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:45.136649 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:45.136649 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:45.136649 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:45.136649 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:45.136649 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:45.136649 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:45.136649 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:45.136649 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:45.136649 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 15:34:45.136649 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:45.136649 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 24 15:34:45.136649 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:45.136649 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 508/985] 10.222.0.102 () {56 vars in 1131 bytes} [Wed Aug 24 15:34:44 2022] GET /identity/v3/services/object-store => generated 93 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:45.140195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d3a44a-3085-45a0-b6bd-6bb1be7f49b4 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:45.247100 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:45.253267 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d3a44a-3085-45a0-b6bd-6bb1be7f49b4 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:45.293763 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2e4d0e-10fb-48f6-9d50-852523291edc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:45.294359 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2e4d0e-10fb-48f6-9d50-852523291edc None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:45.294643 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2e4d0e-10fb-48f6-9d50-852523291edc None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:45.294918 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2e4d0e-10fb-48f6-9d50-852523291edc None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:45.295559 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2e4d0e-10fb-48f6-9d50-852523291edc None admin] RBAC: Adding query filter params (name=object-store) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:45.295884 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2e4d0e-10fb-48f6-9d50-852523291edc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:45.298089 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2e4d0e-10fb-48f6-9d50-852523291edc None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:45.307838 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 478/986] 10.222.0.102 () {56 vars in 1127 bytes} [Wed Aug 24 15:34:45 2022] GET /identity/v3/services?name=object-store => generated 130 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:45.309030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a720a6-e385-4f81-af3e-185d55454353 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:45.421182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a720a6-e385-4f81-af3e-185d55454353 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:45.424222 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:45.460469 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b10bf2d-7956-4d25-8143-e594b0bb84f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:45.461085 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b10bf2d-7956-4d25-8143-e594b0bb84f0 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:45.461369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b10bf2d-7956-4d25-8143-e594b0bb84f0 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:45.461650 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b10bf2d-7956-4d25-8143-e594b0bb84f0 None admin] PATH_INFO: `/v3/services` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:45.462288 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b10bf2d-7956-4d25-8143-e594b0bb84f0 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:45.462594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b10bf2d-7956-4d25-8143-e594b0bb84f0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:45.464000 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b10bf2d-7956-4d25-8143-e594b0bb84f0 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:45.470199 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 509/987] 10.222.0.102 () {56 vars in 1127 bytes} [Wed Aug 24 15:34:45 2022] GET /identity/v3/services?type=object-store => generated 358 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:45.476129 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2e4d0e-10fb-48f6-9d50-852523291edc None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:45.493697 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c24bb87-c62d-4ab5-af27-6848fd62416f None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:45.494170 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c24bb87-c62d-4ab5-af27-6848fd62416f None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:45.494280 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c24bb87-c62d-4ab5-af27-6848fd62416f None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:45.495906 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 510/988] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:45 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:34:45.501826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d0bd87-9d38-4aa4-bd5d-35f3e5177411 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:45.502112 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d0bd87-9d38-4aa4-bd5d-35f3e5177411 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:45.502482 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d0bd87-9d38-4aa4-bd5d-35f3e5177411 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:45.521941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-28d0bd87-9d38-4aa4-bd5d-35f3e5177411 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:45.523161 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-28d0bd87-9d38-4aa4-bd5d-35f3e5177411 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:45.594870 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2e4d0e-10fb-48f6-9d50-852523291edc None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:45.599598 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 511/989] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:45 2022] POST /identity/v3/auth/tokens => generated 1572 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 15:34:45.629866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943244f3-9606-456e-ac03-96e489518331 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:45.630725 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943244f3-9606-456e-ac03-96e489518331 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:45.631168 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943244f3-9606-456e-ac03-96e489518331 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:45.631583 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943244f3-9606-456e-ac03-96e489518331 None admin] PATH_INFO: `/v3/endpoints` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:45.632465 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943244f3-9606-456e-ac03-96e489518331 None admin] RBAC: Adding query filter params (interface=admin, service_id=71e4b2caecb345f694d8317e607bfa38, region_id=RegionOne) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:45.632920 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943244f3-9606-456e-ac03-96e489518331 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:45.634413 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943244f3-9606-456e-ac03-96e489518331 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:45.640039 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 479/990] 10.222.0.102 () {56 vars in 1254 bytes} [Wed Aug 24 15:34:45 2022] GET /identity/v3/endpoints?service_id=71e4b2caecb345f694d8317e607bfa38&interface=admin®ion_id=RegionOne => generated 194 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:45.645816 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d0bd87-9d38-4aa4-bd5d-35f3e5177411 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:45.746681 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:45.752619 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d0bd87-9d38-4aa4-bd5d-35f3e5177411 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:45.795741 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb10aff3-c2b2-4de1-b87e-515e5380cfc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:45.796352 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb10aff3-c2b2-4de1-b87e-515e5380cfc7 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:45.796660 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb10aff3-c2b2-4de1-b87e-515e5380cfc7 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:45.796931 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb10aff3-c2b2-4de1-b87e-515e5380cfc7 None admin] PATH_INFO: `/v3/services` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:45.797570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb10aff3-c2b2-4de1-b87e-515e5380cfc7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:45.798930 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb10aff3-c2b2-4de1-b87e-515e5380cfc7 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:45.811921 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 512/991] 10.222.0.102 () {56 vars in 1092 bytes} [Wed Aug 24 15:34:45 2022] GET /identity/v3/services => generated 1735 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:34:45.873600 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943244f3-9606-456e-ac03-96e489518331 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:45.908662 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:45.996435 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943244f3-9606-456e-ac03-96e489518331 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:46.037510 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0754ed54-5fa8-49a1-bbe7-13cc9ec703f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:46.038052 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0754ed54-5fa8-49a1-bbe7-13cc9ec703f4 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:46.038330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0754ed54-5fa8-49a1-bbe7-13cc9ec703f4 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:46.038604 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0754ed54-5fa8-49a1-bbe7-13cc9ec703f4 None admin] PATH_INFO: `/v3/services/image` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:46.039363 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0754ed54-5fa8-49a1-bbe7-13cc9ec703f4 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:46.041024 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0754ed54-5fa8-49a1-bbe7-13cc9ec703f4 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:46.051031 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-0754ed54-5fa8-49a1-bbe7-13cc9ec703f4 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 24 15:34:46.051031 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:46.051031 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 15:34:46.051031 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 15:34:46.051031 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 15:34:46.051031 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 15:34:46.051031 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 15:34:46.051031 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:46.051031 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 24 15:34:46.051031 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:46.051031 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 15:34:46.051031 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:46.051031 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:46.051031 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:46.051031 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:46.051031 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:46.051031 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:46.052314 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:46.052314 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:46.052314 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:46.052314 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:46.052314 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:46.052314 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:46.052314 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 15:34:46.052314 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 15:34:46.052314 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 15:34:46.052314 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 15:34:46.052314 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:46.052314 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:46.052314 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:46.052314 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:46.052314 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:46.052314 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:46.054027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:46.054027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:46.054027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:46.054027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:46.054027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:46.054027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:46.054027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:46.054027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:46.054027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:46.054027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:46.054027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 15:34:46.054027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:46.054027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 24 15:34:46.054027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:46.054027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 480/992] 10.222.0.102 () {56 vars in 1110 bytes} [Wed Aug 24 15:34:45 2022] GET /identity/v3/services/image => generated 86 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:46.059776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb10aff3-c2b2-4de1-b87e-515e5380cfc7 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:46.205595 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb10aff3-c2b2-4de1-b87e-515e5380cfc7 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:46.244699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2bdc1a-9a42-4f3b-bc76-f95cca49ce36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:46.245316 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2bdc1a-9a42-4f3b-bc76-f95cca49ce36 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:46.245598 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2bdc1a-9a42-4f3b-bc76-f95cca49ce36 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:46.245870 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2bdc1a-9a42-4f3b-bc76-f95cca49ce36 None admin] PATH_INFO: `/v3/services` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:46.246509 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a2bdc1a-9a42-4f3b-bc76-f95cca49ce36 None admin] RBAC: Adding query filter params (name=image) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:46.246813 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a2bdc1a-9a42-4f3b-bc76-f95cca49ce36 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:46.248228 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a2bdc1a-9a42-4f3b-bc76-f95cca49ce36 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:46.254580 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 513/993] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:46 2022] GET /identity/v3/services?name=image => generated 123 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:46.260107 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0754ed54-5fa8-49a1-bbe7-13cc9ec703f4 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:46.362614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:46.374221 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0754ed54-5fa8-49a1-bbe7-13cc9ec703f4 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:46.397379 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbecb5b9-fa17-4fb0-90ed-dfe817d0e6a0 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:46.397694 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbecb5b9-fa17-4fb0-90ed-dfe817d0e6a0 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:46.397934 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbecb5b9-fa17-4fb0-90ed-dfe817d0e6a0 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:46.399396 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 514/994] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:46 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:34:46.404392 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4587b7-5c31-472f-b58f-a956d7a29efd None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:46.404683 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4587b7-5c31-472f-b58f-a956d7a29efd None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:46.404949 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4587b7-5c31-472f-b58f-a956d7a29efd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:46.411807 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aca2413-fe36-446d-8951-e799faaf82d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:46.412414 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aca2413-fe36-446d-8951-e799faaf82d0 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:46.412694 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aca2413-fe36-446d-8951-e799faaf82d0 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:46.412970 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aca2413-fe36-446d-8951-e799faaf82d0 None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:46.413611 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aca2413-fe36-446d-8951-e799faaf82d0 None admin] RBAC: Adding query filter params (type=image) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:46.413915 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aca2413-fe36-446d-8951-e799faaf82d0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:46.415278 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aca2413-fe36-446d-8951-e799faaf82d0 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:46.427885 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 481/995] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:46 2022] GET /identity/v3/services?type=image => generated 123 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:46.428101 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-5a4587b7-5c31-472f-b58f-a956d7a29efd None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:46.428101 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-5a4587b7-5c31-472f-b58f-a956d7a29efd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:46.450933 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:46.508133 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 515/996] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:46 2022] POST /identity/v3/auth/tokens => generated 1572 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 15:34:46.783942 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aca2413-fe36-446d-8951-e799faaf82d0 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:46.900201 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aca2413-fe36-446d-8951-e799faaf82d0 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:46.937810 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0abf6f-a505-4af9-ad3c-e2c1274757a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:46.939165 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0abf6f-a505-4af9-ad3c-e2c1274757a8 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:46.939449 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0abf6f-a505-4af9-ad3c-e2c1274757a8 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:46.939764 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0abf6f-a505-4af9-ad3c-e2c1274757a8 None admin] PATH_INFO: `/v3/services/compute` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:46.940368 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0abf6f-a505-4af9-ad3c-e2c1274757a8 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:46.942059 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0abf6f-a505-4af9-ad3c-e2c1274757a8 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:46.951742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-eb0abf6f-a505-4af9-ad3c-e2c1274757a8 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 24 15:34:46.951742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:46.951742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 15:34:46.951742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 15:34:46.951742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 15:34:46.951742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 15:34:46.951742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 15:34:46.951742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:46.951742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 24 15:34:46.951742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:46.951742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 15:34:46.951742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:46.951742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:46.951742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:46.951742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:46.951742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:46.951742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:46.953176 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:46.953176 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:46.953176 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:46.953176 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:46.953176 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:46.953176 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:46.953176 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 15:34:46.953176 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 15:34:46.953176 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 15:34:46.953176 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 15:34:46.953176 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:46.953176 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:46.953176 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:46.953176 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:46.953176 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:46.953176 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:46.954968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:46.954968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:46.954968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:46.954968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:46.954968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:46.954968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:46.954968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:46.954968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:46.954968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:46.954968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:46.954968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 15:34:46.954968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:46.954968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 24 15:34:46.954968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:46.961700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 482/997] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:34:46 2022] GET /identity/v3/services/compute => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:46.961975 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4587b7-5c31-472f-b58f-a956d7a29efd None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:47.070911 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4587b7-5c31-472f-b58f-a956d7a29efd None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:47.105848 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b8f4a0-4f4b-4b43-b6d2-2fddaea6c0dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:47.106451 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b8f4a0-4f4b-4b43-b6d2-2fddaea6c0dd None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:47.106731 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b8f4a0-4f4b-4b43-b6d2-2fddaea6c0dd None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:47.107004 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b8f4a0-4f4b-4b43-b6d2-2fddaea6c0dd None admin] PATH_INFO: `/v3/services` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:47.107652 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b8f4a0-4f4b-4b43-b6d2-2fddaea6c0dd None admin] RBAC: Adding query filter params (name=compute) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:47.107974 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b8f4a0-4f4b-4b43-b6d2-2fddaea6c0dd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:47.109906 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b8f4a0-4f4b-4b43-b6d2-2fddaea6c0dd None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:47.119332 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 516/998] 10.222.0.102 () {56 vars in 1117 bytes} [Wed Aug 24 15:34:46 2022] GET /identity/v3/services?name=compute => generated 125 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:47.121018 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0abf6f-a505-4af9-ad3c-e2c1274757a8 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:47.141340 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:47.230311 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0abf6f-a505-4af9-ad3c-e2c1274757a8 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:47.266416 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294ece2c-38f2-4629-b7b5-0315fad496ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:47.267050 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294ece2c-38f2-4629-b7b5-0315fad496ec None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:47.267352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294ece2c-38f2-4629-b7b5-0315fad496ec None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:47.267653 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294ece2c-38f2-4629-b7b5-0315fad496ec None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:47.268309 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-294ece2c-38f2-4629-b7b5-0315fad496ec None admin] RBAC: Adding query filter params (type=compute) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:47.268611 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-294ece2c-38f2-4629-b7b5-0315fad496ec None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:47.270104 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-294ece2c-38f2-4629-b7b5-0315fad496ec None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:47.284440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 483/999] 10.222.0.102 () {56 vars in 1117 bytes} [Wed Aug 24 15:34:47 2022] GET /identity/v3/services?type=compute => generated 354 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:47.286084 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b8f4a0-4f4b-4b43-b6d2-2fddaea6c0dd None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:47.310156 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:47.396987 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b8f4a0-4f4b-4b43-b6d2-2fddaea6c0dd None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:47.435804 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ec0ca8-753b-48a3-a68d-db998b3a5ff2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:47.436424 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ec0ca8-753b-48a3-a68d-db998b3a5ff2 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:47.436707 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ec0ca8-753b-48a3-a68d-db998b3a5ff2 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:47.436983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ec0ca8-753b-48a3-a68d-db998b3a5ff2 None admin] PATH_INFO: `/v3/endpoints` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:47.437773 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ec0ca8-753b-48a3-a68d-db998b3a5ff2 None admin] RBAC: Adding query filter params (interface=public, service_id=34fe3adb9abf4b068d55b23f41f6cbbc, region_id=RegionOne) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:47.438084 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ec0ca8-753b-48a3-a68d-db998b3a5ff2 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:47.439459 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ec0ca8-753b-48a3-a68d-db998b3a5ff2 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:47.445433 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 517/1000] 10.222.0.102 () {56 vars in 1256 bytes} [Wed Aug 24 15:34:47 2022] GET /identity/v3/endpoints?service_id=34fe3adb9abf4b068d55b23f41f6cbbc&interface=public®ion_id=RegionOne => generated 195 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:47.451309 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294ece2c-38f2-4629-b7b5-0315fad496ec None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:47.547944 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:47.568795 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294ece2c-38f2-4629-b7b5-0315fad496ec None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:47.603609 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029f0419-3d0c-4038-bb84-399e2d2b5542 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:47.604226 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029f0419-3d0c-4038-bb84-399e2d2b5542 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:47.604501 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029f0419-3d0c-4038-bb84-399e2d2b5542 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:47.604770 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029f0419-3d0c-4038-bb84-399e2d2b5542 None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:47.605372 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029f0419-3d0c-4038-bb84-399e2d2b5542 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:47.606726 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029f0419-3d0c-4038-bb84-399e2d2b5542 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:47.619878 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 484/1001] 10.222.0.102 () {56 vars in 1092 bytes} [Wed Aug 24 15:34:47 2022] GET /identity/v3/services => generated 1735 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:34:47.827674 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf65dda8-0d73-4ec8-b50a-c3f87c9727f7 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:47.828446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf65dda8-0d73-4ec8-b50a-c3f87c9727f7 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:47.828834 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf65dda8-0d73-4ec8-b50a-c3f87c9727f7 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:47.830417 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 518/1002] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:47 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:47.835741 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa3c880-92a2-45a6-9bb6-45ceac16c09a None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:47.836032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa3c880-92a2-45a6-9bb6-45ceac16c09a None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:47.836303 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa3c880-92a2-45a6-9bb6-45ceac16c09a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:47.859820 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-afa3c880-92a2-45a6-9bb6-45ceac16c09a None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:47.861038 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-afa3c880-92a2-45a6-9bb6-45ceac16c09a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:47.947682 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 485/1003] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:47 2022] POST /identity/v3/auth/tokens => generated 1572 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:34:48.213618 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf65dda8-0d73-4ec8-b50a-c3f87c9727f7 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:48.304146 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:48.322940 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf65dda8-0d73-4ec8-b50a-c3f87c9727f7 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:48.357926 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-927b1543-67e6-48e3-b31b-e1ae2f02313f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:48.358549 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927b1543-67e6-48e3-b31b-e1ae2f02313f None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:48.358823 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927b1543-67e6-48e3-b31b-e1ae2f02313f None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:48.359646 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927b1543-67e6-48e3-b31b-e1ae2f02313f None admin] PATH_INFO: `/v3/services/object-store` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:48.360251 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-927b1543-67e6-48e3-b31b-e1ae2f02313f None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:48.361613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-927b1543-67e6-48e3-b31b-e1ae2f02313f None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:48.370609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-927b1543-67e6-48e3-b31b-e1ae2f02313f None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 24 15:34:48.370609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:48.370609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 15:34:48.370609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 15:34:48.370609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 15:34:48.370609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 15:34:48.370609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 15:34:48.370609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:48.370609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 24 15:34:48.370609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:48.370609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 15:34:48.370609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:48.370609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:48.370609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:48.370609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:48.370609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:48.370609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:48.371617 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:48.371617 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:48.371617 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:48.371617 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:48.371617 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:48.371617 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:48.371617 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 15:34:48.371617 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 15:34:48.371617 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 15:34:48.371617 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 15:34:48.371617 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:48.371617 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:48.371617 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:48.371617 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:48.371617 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:48.371617 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:48.373039 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:48.373039 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:48.373039 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:48.373039 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:48.373039 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:48.373039 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:48.373039 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:48.373039 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:48.373039 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:48.373039 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:48.373039 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 15:34:48.373039 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:48.373039 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 24 15:34:48.373039 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:48.374130 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 519/1004] 10.222.0.102 () {56 vars in 1131 bytes} [Wed Aug 24 15:34:48 2022] GET /identity/v3/services/object-store => generated 93 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:48.378239 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa3c880-92a2-45a6-9bb6-45ceac16c09a None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:48.399665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b94503-7237-4383-942b-a392781eb8b1 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:48.399665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b94503-7237-4383-942b-a392781eb8b1 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:48.399665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b94503-7237-4383-942b-a392781eb8b1 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:48.399665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 520/1005] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:48 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:48.405129 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0682dd2-9b3b-4e9e-bc2f-ba3fa5b9d6c0 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:48.405463 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0682dd2-9b3b-4e9e-bc2f-ba3fa5b9d6c0 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:48.405698 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0682dd2-9b3b-4e9e-bc2f-ba3fa5b9d6c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:48.425755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-c0682dd2-9b3b-4e9e-bc2f-ba3fa5b9d6c0 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:48.426957 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-c0682dd2-9b3b-4e9e-bc2f-ba3fa5b9d6c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:48.473251 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:48.504031 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 521/1006] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:48 2022] POST /identity/v3/auth/tokens => generated 1572 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 15:34:48.507217 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa3c880-92a2-45a6-9bb6-45ceac16c09a None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:48.545979 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed76d6a-8250-4891-9f96-bea2d612587c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:48.546590 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed76d6a-8250-4891-9f96-bea2d612587c None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:48.546867 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed76d6a-8250-4891-9f96-bea2d612587c None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:48.547462 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed76d6a-8250-4891-9f96-bea2d612587c None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:48.547811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ed76d6a-8250-4891-9f96-bea2d612587c None admin] RBAC: Adding query filter params (name=object-store) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:48.548113 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ed76d6a-8250-4891-9f96-bea2d612587c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:48.549461 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ed76d6a-8250-4891-9f96-bea2d612587c None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:48.559109 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 486/1007] 10.222.0.102 () {56 vars in 1127 bytes} [Wed Aug 24 15:34:48 2022] GET /identity/v3/services?name=object-store => generated 130 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:48.560668 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0682dd2-9b3b-4e9e-bc2f-ba3fa5b9d6c0 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:48.647735 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:48.675536 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0682dd2-9b3b-4e9e-bc2f-ba3fa5b9d6c0 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:48.715735 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d83f07-9251-4046-8dfb-9f4adab622bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:48.716653 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d83f07-9251-4046-8dfb-9f4adab622bb None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:48.717097 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d83f07-9251-4046-8dfb-9f4adab622bb None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:48.717536 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d83f07-9251-4046-8dfb-9f4adab622bb None admin] PATH_INFO: `/v3/services` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:48.718432 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d83f07-9251-4046-8dfb-9f4adab622bb None admin] RBAC: Adding query filter params (type=object-store) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:48.719719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d83f07-9251-4046-8dfb-9f4adab622bb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:48.721292 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d83f07-9251-4046-8dfb-9f4adab622bb None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:48.727393 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 522/1008] 10.222.0.102 () {56 vars in 1127 bytes} [Wed Aug 24 15:34:48 2022] GET /identity/v3/services?type=object-store => generated 358 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:48.732812 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed76d6a-8250-4891-9f96-bea2d612587c None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:48.773073 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d83f07-9251-4046-8dfb-9f4adab622bb None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:48.849371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed76d6a-8250-4891-9f96-bea2d612587c None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:48.885209 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac57e8b3-c434-4909-91f6-cc11a4d280a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:48.885812 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac57e8b3-c434-4909-91f6-cc11a4d280a5 None admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:48.886089 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac57e8b3-c434-4909-91f6-cc11a4d280a5 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:48.886359 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac57e8b3-c434-4909-91f6-cc11a4d280a5 None admin] PATH_INFO: `/v3/endpoints` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:48.887025 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac57e8b3-c434-4909-91f6-cc11a4d280a5 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:48.888427 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac57e8b3-c434-4909-91f6-cc11a4d280a5 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:48.900875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:48.904941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d83f07-9251-4046-8dfb-9f4adab622bb None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:48.908511 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 487/1009] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:34:48 2022] POST /identity/v3/endpoints => generated 327 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 15:34:48.955803 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2729f720-3cba-4636-a91f-ab066b481659 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:48.956804 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2729f720-3cba-4636-a91f-ab066b481659 None admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:48.957263 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2729f720-3cba-4636-a91f-ab066b481659 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:48.957752 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2729f720-3cba-4636-a91f-ab066b481659 None admin] PATH_INFO: `/v3/services` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:48.958614 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2729f720-3cba-4636-a91f-ab066b481659 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:48.960593 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2729f720-3cba-4636-a91f-ab066b481659 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:48.972752 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:48.985179 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 523/1010] 10.222.0.102 () {60 vars in 1146 bytes} [Wed Aug 24 15:34:48 2022] POST /identity/v3/services => generated 243 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 15:34:49.622762 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb008e01-0c5c-4c74-8177-b0022f87a7fb None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:49.623321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb008e01-0c5c-4c74-8177-b0022f87a7fb None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:49.623321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb008e01-0c5c-4c74-8177-b0022f87a7fb None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:49.625415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 488/1011] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:49 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:49.631180 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73d5ca7-3b70-4907-a8e9-893e1cd055e6 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:49.631616 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73d5ca7-3b70-4907-a8e9-893e1cd055e6 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:49.631855 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73d5ca7-3b70-4907-a8e9-893e1cd055e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:49.651498 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-f73d5ca7-3b70-4907-a8e9-893e1cd055e6 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:49.652731 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-f73d5ca7-3b70-4907-a8e9-893e1cd055e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:49.741237 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 524/1012] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:49 2022] POST /identity/v3/auth/tokens => generated 1816 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:34:50.022626 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb008e01-0c5c-4c74-8177-b0022f87a7fb None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:50.142332 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb008e01-0c5c-4c74-8177-b0022f87a7fb None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:50.180630 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ddfcdf-0746-4f96-a931-f3a4e3dd138f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:50.181260 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ddfcdf-0746-4f96-a931-f3a4e3dd138f None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:50.181549 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ddfcdf-0746-4f96-a931-f3a4e3dd138f None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:50.181822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ddfcdf-0746-4f96-a931-f3a4e3dd138f None admin] PATH_INFO: `/v3/services/compute` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:50.182418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ddfcdf-0746-4f96-a931-f3a4e3dd138f None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:50.183819 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ddfcdf-0746-4f96-a931-f3a4e3dd138f None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:50.194041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-b6ddfcdf-0746-4f96-a931-f3a4e3dd138f None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 24 15:34:50.194041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:50.194041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 15:34:50.194041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 15:34:50.194041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 15:34:50.194041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 15:34:50.194041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 15:34:50.194041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:50.194041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 24 15:34:50.194041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:50.194041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 15:34:50.194041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:50.194041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:50.194041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:50.194041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:50.194041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:50.194041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:50.195088 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:50.195088 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:50.195088 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:50.195088 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:50.195088 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:50.195088 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:50.195088 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 15:34:50.195088 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 15:34:50.195088 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 15:34:50.195088 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 15:34:50.195088 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:50.195088 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:50.195088 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:50.195088 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:50.195088 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:50.195088 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:50.195986 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:50.195986 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:50.195986 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:50.195986 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:50.195986 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:50.195986 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:50.195986 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:50.195986 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:50.195986 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:50.195986 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:50.195986 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 15:34:50.195986 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:50.195986 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 24 15:34:50.195986 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:50.195986 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 489/1013] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:34:50 2022] GET /identity/v3/services/compute => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:50.201224 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73d5ca7-3b70-4907-a8e9-893e1cd055e6 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:50.305696 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73d5ca7-3b70-4907-a8e9-893e1cd055e6 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:50.329523 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:50.341197 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab0e957-2c7d-43bb-aaf4-7beb9a54bb67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:50.341806 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab0e957-2c7d-43bb-aaf4-7beb9a54bb67 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:50.342086 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab0e957-2c7d-43bb-aaf4-7beb9a54bb67 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:50.342420 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab0e957-2c7d-43bb-aaf4-7beb9a54bb67 None admin] PATH_INFO: `/v3/services` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:50.343035 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab0e957-2c7d-43bb-aaf4-7beb9a54bb67 None admin] RBAC: Adding query filter params (name=compute) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:50.343287 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab0e957-2c7d-43bb-aaf4-7beb9a54bb67 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:50.345308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab0e957-2c7d-43bb-aaf4-7beb9a54bb67 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:50.355026 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 525/1014] 10.222.0.102 () {56 vars in 1117 bytes} [Wed Aug 24 15:34:50 2022] GET /identity/v3/services?name=compute => generated 125 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:50.356602 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ddfcdf-0746-4f96-a931-f3a4e3dd138f None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:50.424926 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:50.466731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ddfcdf-0746-4f96-a931-f3a4e3dd138f None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:50.503579 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9fce26-0574-4894-a297-0ec9b2370e09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:50.504214 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9fce26-0574-4894-a297-0ec9b2370e09 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:50.504556 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9fce26-0574-4894-a297-0ec9b2370e09 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:50.504834 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9fce26-0574-4894-a297-0ec9b2370e09 None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:50.505468 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9fce26-0574-4894-a297-0ec9b2370e09 None admin] RBAC: Adding query filter params (type=compute) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:50.505770 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9fce26-0574-4894-a297-0ec9b2370e09 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:50.507135 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9fce26-0574-4894-a297-0ec9b2370e09 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:50.512539 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 490/1015] 10.222.0.102 () {56 vars in 1117 bytes} [Wed Aug 24 15:34:50 2022] GET /identity/v3/services?type=compute => generated 354 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:50.518405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab0e957-2c7d-43bb-aaf4-7beb9a54bb67 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:50.623518 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab0e957-2c7d-43bb-aaf4-7beb9a54bb67 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:50.661766 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4281c591-5f07-48e8-a48b-1ad98c6cd3cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:50.662539 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4281c591-5f07-48e8-a48b-1ad98c6cd3cc None admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:50.662665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4281c591-5f07-48e8-a48b-1ad98c6cd3cc None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:50.662950 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4281c591-5f07-48e8-a48b-1ad98c6cd3cc None admin] PATH_INFO: `/v3/endpoints` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:50.663512 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4281c591-5f07-48e8-a48b-1ad98c6cd3cc None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:50.664926 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4281c591-5f07-48e8-a48b-1ad98c6cd3cc None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:50.695673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 526/1016] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:34:50 2022] POST /identity/v3/endpoints => generated 336 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:34:50.886635 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a377f5-820a-4971-8ff1-52ea017dfcf2 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:50.887147 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a377f5-820a-4971-8ff1-52ea017dfcf2 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:50.887257 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a377f5-820a-4971-8ff1-52ea017dfcf2 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:50.888553 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 491/1017] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:50 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:50.894375 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5467b948-dd12-41f8-b4d2-72577073ef15 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:50.894665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5467b948-dd12-41f8-b4d2-72577073ef15 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:50.895007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5467b948-dd12-41f8-b4d2-72577073ef15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:50.914150 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-5467b948-dd12-41f8-b4d2-72577073ef15 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:50.915353 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-5467b948-dd12-41f8-b4d2-72577073ef15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:50.961508 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:50.972100 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1327919-d8eb-4f42-8fd2-a6dc19949ac2 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:50.972406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1327919-d8eb-4f42-8fd2-a6dc19949ac2 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:50.972663 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1327919-d8eb-4f42-8fd2-a6dc19949ac2 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:50.974311 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 492/1018] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:50 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:34:50.979767 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8922c8cb-af6c-4380-9c9f-ed0e3d66baf0 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:50.980058 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8922c8cb-af6c-4380-9c9f-ed0e3d66baf0 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:50.980335 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8922c8cb-af6c-4380-9c9f-ed0e3d66baf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:50.998915 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 527/1019] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:50 2022] POST /identity/v3/auth/tokens => generated 1973 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:34:51.002473 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-8922c8cb-af6c-4380-9c9f-ed0e3d66baf0 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:51.003722 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-8922c8cb-af6c-4380-9c9f-ed0e3d66baf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:51.078478 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 493/1020] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:50 2022] POST /identity/v3/auth/tokens => generated 1973 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 15:34:51.264776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5467b948-dd12-41f8-b4d2-72577073ef15 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:51.349377 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8922c8cb-af6c-4380-9c9f-ed0e3d66baf0 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:51.374772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5467b948-dd12-41f8-b4d2-72577073ef15 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:51.412429 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:51.420606 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3d7c3c-e358-458d-ac3a-e7017ca20143 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:51.421502 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3d7c3c-e358-458d-ac3a-e7017ca20143 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:51.421935 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3d7c3c-e358-458d-ac3a-e7017ca20143 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:51.422677 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3d7c3c-e358-458d-ac3a-e7017ca20143 None admin] PATH_INFO: `/v3/domains/default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:51.424383 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3d7c3c-e358-458d-ac3a-e7017ca20143 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:51.426104 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3d7c3c-e358-458d-ac3a-e7017ca20143 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:51.433328 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 528/1021] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:51 2022] GET /identity/v3/domains/default => generated 200 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:51.434755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3d7c3c-e358-458d-ac3a-e7017ca20143 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:51.461971 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8922c8cb-af6c-4380-9c9f-ed0e3d66baf0 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:51.482770 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:51.507022 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449cce0e-57d0-4c00-a329-71fed181490f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:51.507672 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449cce0e-57d0-4c00-a329-71fed181490f None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:51.507957 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449cce0e-57d0-4c00-a329-71fed181490f None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:51.508230 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449cce0e-57d0-4c00-a329-71fed181490f None admin] PATH_INFO: `/v3/services/image` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:51.508805 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449cce0e-57d0-4c00-a329-71fed181490f None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:51.510169 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449cce0e-57d0-4c00-a329-71fed181490f None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:51.520728 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-449cce0e-57d0-4c00-a329-71fed181490f None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 24 15:34:51.520728 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:51.520728 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 15:34:51.520728 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 15:34:51.520728 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 15:34:51.520728 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 15:34:51.520728 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 15:34:51.520728 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:51.520728 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 24 15:34:51.520728 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:51.520728 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 15:34:51.520728 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:51.520728 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:51.520728 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:51.520728 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:51.520728 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:51.520728 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:51.521738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:51.521738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:51.521738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:51.521738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:51.521738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:51.521738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:51.521738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 15:34:51.521738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 15:34:51.521738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 15:34:51.521738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 15:34:51.521738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:51.521738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:51.521738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:51.521738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:51.521738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:51.521738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:51.524709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:51.524709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:51.524709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:51.524709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:51.524709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:51.524709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:51.524709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:51.524709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:51.524709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:51.524709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:51.524709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 15:34:51.524709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:51.524709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 24 15:34:51.524709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:51.524709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 494/1022] 10.222.0.102 () {56 vars in 1110 bytes} [Wed Aug 24 15:34:51 2022] GET /identity/v3/services/image => generated 86 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:34:51.527530 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449cce0e-57d0-4c00-a329-71fed181490f None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:51.563152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3d7c3c-e358-458d-ac3a-e7017ca20143 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:51.604189 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070e5d51-3bec-4879-a1dd-701c6fecf2ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:51.604802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070e5d51-3bec-4879-a1dd-701c6fecf2ff None admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:51.605085 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070e5d51-3bec-4879-a1dd-701c6fecf2ff None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:51.605362 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070e5d51-3bec-4879-a1dd-701c6fecf2ff None admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:51.606014 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070e5d51-3bec-4879-a1dd-701c6fecf2ff None admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:51.607447 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070e5d51-3bec-4879-a1dd-701c6fecf2ff None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:51.637288 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 529/1023] 10.222.0.102 () {60 vars in 1146 bytes} [Wed Aug 24 15:34:51 2022] POST /identity/v3/projects => generated 312 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 15:34:51.661387 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449cce0e-57d0-4c00-a329-71fed181490f None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:51.692193 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:51.706183 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f439efb1-021c-4fd0-9a65-d2ee569f2ac0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:51.706773 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f439efb1-021c-4fd0-9a65-d2ee569f2ac0 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:51.707055 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f439efb1-021c-4fd0-9a65-d2ee569f2ac0 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:51.707399 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f439efb1-021c-4fd0-9a65-d2ee569f2ac0 None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:51.708090 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f439efb1-021c-4fd0-9a65-d2ee569f2ac0 None admin] RBAC: Adding query filter params (name=image) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:51.708398 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f439efb1-021c-4fd0-9a65-d2ee569f2ac0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:51.709789 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f439efb1-021c-4fd0-9a65-d2ee569f2ac0 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:51.721239 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 495/1024] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:51 2022] GET /identity/v3/services?name=image => generated 123 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:51.723220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070e5d51-3bec-4879-a1dd-701c6fecf2ff None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:51.838473 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:51.857294 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070e5d51-3bec-4879-a1dd-701c6fecf2ff None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:51.896290 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f54684-9e24-49a5-a81a-6b822ff2597c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:51.896909 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f54684-9e24-49a5-a81a-6b822ff2597c None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:51.897177 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f54684-9e24-49a5-a81a-6b822ff2597c None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:51.897451 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f54684-9e24-49a5-a81a-6b822ff2597c None admin] PATH_INFO: `/v3/services` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:51.898079 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f54684-9e24-49a5-a81a-6b822ff2597c None admin] RBAC: Adding query filter params (type=image) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:51.898382 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f54684-9e24-49a5-a81a-6b822ff2597c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:51.899749 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f54684-9e24-49a5-a81a-6b822ff2597c None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:51.905136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 530/1025] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:51 2022] GET /identity/v3/services?type=image => generated 352 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:51.911242 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f439efb1-021c-4fd0-9a65-d2ee569f2ac0 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:52.029886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f439efb1-021c-4fd0-9a65-d2ee569f2ac0 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:52.066183 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a166c9-3e05-425c-bb10-054267734268 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:52.067045 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a166c9-3e05-425c-bb10-054267734268 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:52.067322 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a166c9-3e05-425c-bb10-054267734268 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:52.067592 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a166c9-3e05-425c-bb10-054267734268 None admin] PATH_INFO: `/v3/endpoints` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:52.068328 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a166c9-3e05-425c-bb10-054267734268 None admin] RBAC: Adding query filter params (interface=public, service_id=63133c60f487450aa4b130b0d142e264, region_id=RegionOne) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:52.068640 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a166c9-3e05-425c-bb10-054267734268 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:52.070001 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a166c9-3e05-425c-bb10-054267734268 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:52.075432 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 496/1026] 10.222.0.102 () {56 vars in 1256 bytes} [Wed Aug 24 15:34:51 2022] GET /identity/v3/endpoints?service_id=63133c60f487450aa4b130b0d142e264&interface=public®ion_id=RegionOne => generated 195 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:52.081896 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f54684-9e24-49a5-a81a-6b822ff2597c None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:52.191261 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f54684-9e24-49a5-a81a-6b822ff2597c None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:52.226505 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371cdc11-47ad-422b-a87f-305dce4c916c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:52.227119 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371cdc11-47ad-422b-a87f-305dce4c916c None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:52.227402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371cdc11-47ad-422b-a87f-305dce4c916c None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:52.227704 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371cdc11-47ad-422b-a87f-305dce4c916c None admin] PATH_INFO: `/v3/services` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:52.228325 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371cdc11-47ad-422b-a87f-305dce4c916c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:52.229688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371cdc11-47ad-422b-a87f-305dce4c916c None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:52.237142 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 531/1027] 10.222.0.102 () {56 vars in 1092 bytes} [Wed Aug 24 15:34:52 2022] GET /identity/v3/services => generated 1966 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:34:52.810380 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16aaeb5-2a87-481b-afa3-2a39265866dc None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:52.811216 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16aaeb5-2a87-481b-afa3-2a39265866dc None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:52.811216 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16aaeb5-2a87-481b-afa3-2a39265866dc None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:52.813061 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 497/1028] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:52 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:52.819832 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e7c232-6451-4f1b-b08b-f2e86646a68a None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:52.820370 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e7c232-6451-4f1b-b08b-f2e86646a68a None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:52.820443 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e7c232-6451-4f1b-b08b-f2e86646a68a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:52.840057 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-82e7c232-6451-4f1b-b08b-f2e86646a68a None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:52.844510 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-82e7c232-6451-4f1b-b08b-f2e86646a68a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:52.856812 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:52.917390 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 532/1029] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:52 2022] POST /identity/v3/auth/tokens => generated 1973 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:34:53.194861 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16aaeb5-2a87-481b-afa3-2a39265866dc None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:53.256454 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:53.313413 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16aaeb5-2a87-481b-afa3-2a39265866dc None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:53.354103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80868349-aefa-449b-9288-e5e96d6a9219 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:53.354739 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80868349-aefa-449b-9288-e5e96d6a9219 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:53.355183 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80868349-aefa-449b-9288-e5e96d6a9219 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:53.355306 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80868349-aefa-449b-9288-e5e96d6a9219 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:53.366144 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80868349-aefa-449b-9288-e5e96d6a9219 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:34:53.370837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-80868349-aefa-449b-9288-e5e96d6a9219 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 24 15:34:53.370837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:53.370837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:53.370837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:53.370837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:53.370837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:53.370837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:53.370837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:53.370837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:53.370837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:53.370837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:53.370837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:53.370837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:34:53.370837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:34:53.370837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:34:53.370837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:34:53.372000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:34:53.372000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:34:53.372000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:34:53.372000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:34:53.372000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:53.372000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:53.372000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:53.372000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:53.372000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:53.372000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:53.372000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:53.372000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:53.372000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:53.372000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:53.372000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:53.372000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:53.372849 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:53.372849 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:53.372849 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:53.372849 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:53.372849 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:34:53.372849 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:34:53.372849 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:34:53.372849 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:34:53.372849 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:34:53.372849 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:34:53.372849 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 24 15:34:53.372849 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:53.372849 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 498/1030] 10.222.0.102 () {56 vars in 1125 bytes} [Wed Aug 24 15:34:53 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:34:53.380367 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e7c232-6451-4f1b-b08b-f2e86646a68a None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:53.481545 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e7c232-6451-4f1b-b08b-f2e86646a68a None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:53.516531 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17bb694-18e3-40d3-abe7-4f98e1376679 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:53.517172 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17bb694-18e3-40d3-abe7-4f98e1376679 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:53.517460 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17bb694-18e3-40d3-abe7-4f98e1376679 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:53.517780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17bb694-18e3-40d3-abe7-4f98e1376679 None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:53.518419 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e17bb694-18e3-40d3-abe7-4f98e1376679 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:53.518723 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e17bb694-18e3-40d3-abe7-4f98e1376679 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:53.520138 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e17bb694-18e3-40d3-abe7-4f98e1376679 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:53.533511 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 533/1031] 10.222.0.102 () {56 vars in 1120 bytes} [Wed Aug 24 15:34:53 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 340 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:53.535000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80868349-aefa-449b-9288-e5e96d6a9219 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:53.638746 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80868349-aefa-449b-9288-e5e96d6a9219 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:53.650531 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:53.667852 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6875583-b80e-4ef5-9bcd-45bc5323daa0 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:53.668094 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6875583-b80e-4ef5-9bcd-45bc5323daa0 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:53.668437 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6875583-b80e-4ef5-9bcd-45bc5323daa0 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:53.670051 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 534/1032] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:53 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:34:53.676932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deabbf90-7c0f-40a7-afeb-533a0c2bbbbb None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:53.677217 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deabbf90-7c0f-40a7-afeb-533a0c2bbbbb None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:53.677566 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deabbf90-7c0f-40a7-afeb-533a0c2bbbbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:53.678942 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3d31a1-4a1f-45b1-ba77-e0ba7b1f445f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:53.679577 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3d31a1-4a1f-45b1-ba77-e0ba7b1f445f None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:53.679885 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3d31a1-4a1f-45b1-ba77-e0ba7b1f445f None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:53.680159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3d31a1-4a1f-45b1-ba77-e0ba7b1f445f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:53.692074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3d31a1-4a1f-45b1-ba77-e0ba7b1f445f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:53.693610 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3d31a1-4a1f-45b1-ba77-e0ba7b1f445f None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:53.698579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-deabbf90-7c0f-40a7-afeb-533a0c2bbbbb None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:53.699853 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-deabbf90-7c0f-40a7-afeb-533a0c2bbbbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:53.703343 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:53.711082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-dd3d31a1-4a1f-45b1-ba77-e0ba7b1f445f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:53.711082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:53.711082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:53.711082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:53.711082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:53.711082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:53.711082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:53.711082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:53.711082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:53.711082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:53.711082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:53.711082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:53.711082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:53.711082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:53.711082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:53.711082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:53.712295 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:53.712295 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:53.712295 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:53.712295 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:53.712295 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:53.712295 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:53.712295 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:53.712295 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:53.712295 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:53.712295 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:53.712295 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:53.712295 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:53.712295 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:53.712295 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:53.712295 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:53.712295 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:53.712295 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:53.713439 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:53.713439 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:53.713439 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:53.713439 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 499/1033] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:34:53.718504 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3d31a1-4a1f-45b1-ba77-e0ba7b1f445f None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:53.783882 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 535/1034] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:53 2022] POST /identity/v3/auth/tokens => generated 1973 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 15:34:53.836089 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3d31a1-4a1f-45b1-ba77-e0ba7b1f445f None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:53.872491 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4e0ff5-d7dc-411a-b1fe-a59f9ffed003 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:53.873087 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4e0ff5-d7dc-411a-b1fe-a59f9ffed003 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:53.873364 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4e0ff5-d7dc-411a-b1fe-a59f9ffed003 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:53.873633 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4e0ff5-d7dc-411a-b1fe-a59f9ffed003 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:53.874274 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4e0ff5-d7dc-411a-b1fe-a59f9ffed003 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:53.874600 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4e0ff5-d7dc-411a-b1fe-a59f9ffed003 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:53.876212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4e0ff5-d7dc-411a-b1fe-a59f9ffed003 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:53.894689 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 500/1035] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:53 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:53.896552 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deabbf90-7c0f-40a7-afeb-533a0c2bbbbb None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:54.020185 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deabbf90-7c0f-40a7-afeb-533a0c2bbbbb None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:54.055133 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c327db-f20c-4433-8dfb-30c9d8ded0c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:54.055714 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c327db-f20c-4433-8dfb-30c9d8ded0c9 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:54.056000 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c327db-f20c-4433-8dfb-30c9d8ded0c9 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:54.056273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c327db-f20c-4433-8dfb-30c9d8ded0c9 None admin] PATH_INFO: `/v3/users/nova` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:54.060418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4e0ff5-d7dc-411a-b1fe-a59f9ffed003 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:54.065205 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c327db-f20c-4433-8dfb-30c9d8ded0c9 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:54.066933 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c327db-f20c-4433-8dfb-30c9d8ded0c9 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:54.069804 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:54.082025 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-c2c327db-f20c-4433-8dfb-30c9d8ded0c9 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 24 15:34:54.082025 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:54.082025 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:54.082025 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:54.082025 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:54.082025 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:54.082025 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:54.082025 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:54.082025 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:54.082025 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:54.082025 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:54.082025 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:54.082025 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:34:54.082025 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:34:54.082025 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:34:54.082025 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:34:54.083306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:54.083306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:54.083306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:34:54.083306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:54.083306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:34:54.083306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:54.083306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:54.083306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:54.083306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:54.083306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:54.083306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:54.083306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:54.083306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:54.083306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:54.083306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:54.083306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:54.083306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:54.083306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:54.084521 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:54.084521 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:54.084521 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:34:54.084521 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:34:54.084521 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:34:54.084521 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:34:54.084521 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:34:54.084521 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:34:54.084521 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 24 15:34:54.084521 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:54.087982 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 536/1036] 10.222.0.102 () {56 vars in 1098 bytes} [Wed Aug 24 15:34:53 2022] GET /identity/v3/users/nova => generated 82 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:34:54.089171 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c327db-f20c-4433-8dfb-30c9d8ded0c9 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:54.094467 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:54.178714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4e0ff5-d7dc-411a-b1fe-a59f9ffed003 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:54.204227 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c327db-f20c-4433-8dfb-30c9d8ded0c9 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:54.218344 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6513483-a6bf-436b-838e-4125256a148b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:54.219160 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6513483-a6bf-436b-838e-4125256a148b None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:54.219552 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6513483-a6bf-436b-838e-4125256a148b None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:54.219854 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6513483-a6bf-436b-838e-4125256a148b None admin] PATH_INFO: `/v3/domains/default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:54.220987 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6513483-a6bf-436b-838e-4125256a148b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:54.222418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6513483-a6bf-436b-838e-4125256a148b None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:54.224022 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 501/1037] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:54 2022] GET /identity/v3/domains/default => generated 200 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:54.225530 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2688cc32-4aa3-4762-b576-95806dd3ecda None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:54.225809 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2688cc32-4aa3-4762-b576-95806dd3ecda None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:54.226079 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2688cc32-4aa3-4762-b576-95806dd3ecda None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:54.227259 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 502/1038] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:54 2022] GET /identity => generated 271 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:34:54.229511 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2688cc32-4aa3-4762-b576-95806dd3ecda None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:54.249153 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8162c35c-0a93-48ca-99f6-d3f5e1349a9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:54.249762 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8162c35c-0a93-48ca-99f6-d3f5e1349a9e None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:54.250533 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8162c35c-0a93-48ca-99f6-d3f5e1349a9e None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:54.250814 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8162c35c-0a93-48ca-99f6-d3f5e1349a9e None admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:54.251670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8162c35c-0a93-48ca-99f6-d3f5e1349a9e None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:54.252023 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8162c35c-0a93-48ca-99f6-d3f5e1349a9e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:54.253496 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8162c35c-0a93-48ca-99f6-d3f5e1349a9e None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:54.273737 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 537/1039] 10.222.0.102 () {56 vars in 1138 bytes} [Wed Aug 24 15:34:54 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 370 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:54.275157 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9195a111-1948-4265-bb67-b89c3af17f05 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:54.275457 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9195a111-1948-4265-bb67-b89c3af17f05 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:54.275767 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9195a111-1948-4265-bb67-b89c3af17f05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:54.294158 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-9195a111-1948-4265-bb67-b89c3af17f05 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:54.295359 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-9195a111-1948-4265-bb67-b89c3af17f05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:54.335093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:54.346975 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2688cc32-4aa3-4762-b576-95806dd3ecda None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:54.350248 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:54.365989 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 538/1040] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:54 2022] POST /identity/v3/auth/tokens => generated 1973 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:34:54.366719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9195a111-1948-4265-bb67-b89c3af17f05 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:54.386761 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffb22d3-5423-4b15-93ba-d0f940384698 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:54.387689 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffb22d3-5423-4b15-93ba-d0f940384698 None admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:54.388184 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffb22d3-5423-4b15-93ba-d0f940384698 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:54.388752 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffb22d3-5423-4b15-93ba-d0f940384698 None admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:54.389410 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffb22d3-5423-4b15-93ba-d0f940384698 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:54.390849 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffb22d3-5423-4b15-93ba-d0f940384698 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:54.427665 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 503/1041] 10.222.0.102 () {60 vars in 1137 bytes} [Wed Aug 24 15:34:54 2022] POST /identity/v3/users => generated 286 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 15:34:54.482692 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9195a111-1948-4265-bb67-b89c3af17f05 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:54.523269 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90eb22ef-345a-4cde-80e5-c6d17e9a6ff5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:54.523934 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90eb22ef-345a-4cde-80e5-c6d17e9a6ff5 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:54.524218 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90eb22ef-345a-4cde-80e5-c6d17e9a6ff5 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:54.524491 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90eb22ef-345a-4cde-80e5-c6d17e9a6ff5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:54.589079 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:54.603222 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90eb22ef-345a-4cde-80e5-c6d17e9a6ff5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:54.604749 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90eb22ef-345a-4cde-80e5-c6d17e9a6ff5 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:54.639734 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffb22d3-5423-4b15-93ba-d0f940384698 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:54.715788 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 539/1042] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:34:54 2022] GET /identity/v3/auth/tokens => generated 1973 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 15:34:54.721966 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90eb22ef-345a-4cde-80e5-c6d17e9a6ff5 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:54.755041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffb22d3-5423-4b15-93ba-d0f940384698 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:54.791390 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f15c94-7892-4689-87ad-3a33e8593329 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:54.792154 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f15c94-7892-4689-87ad-3a33e8593329 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:54.792505 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f15c94-7892-4689-87ad-3a33e8593329 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:54.792785 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f15c94-7892-4689-87ad-3a33e8593329 None admin] PATH_INFO: `/v3/services/image` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:54.793380 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f15c94-7892-4689-87ad-3a33e8593329 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:54.794810 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f15c94-7892-4689-87ad-3a33e8593329 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:54.804704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-31f15c94-7892-4689-87ad-3a33e8593329 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 24 15:34:54.804704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:54.804704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 15:34:54.804704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 15:34:54.804704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 15:34:54.804704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 15:34:54.804704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 15:34:54.804704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:54.804704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 24 15:34:54.804704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:54.804704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 15:34:54.804704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:54.804704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:54.804704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:54.804704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:54.804704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:54.804704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:54.806992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:54.806992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:54.806992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:54.806992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:54.806992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:54.806992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:54.806992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 15:34:54.806992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 15:34:54.806992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 15:34:54.806992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 15:34:54.806992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:54.806992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:54.806992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:54.806992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:54.806992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:54.806992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:54.808030 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:54.808030 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:54.808030 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:54.808030 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:54.808030 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:54.808030 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:54.808030 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:54.808030 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:54.808030 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:54.808030 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:54.808030 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 15:34:54.808030 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:34:54.808030 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 24 15:34:54.808030 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:54.808030 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 504/1043] 10.222.0.102 () {56 vars in 1110 bytes} [Wed Aug 24 15:34:54 2022] GET /identity/v3/services/image => generated 86 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:54.811838 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f15c94-7892-4689-87ad-3a33e8593329 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:54.824696 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:54.830229 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:54.841040 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90eb22ef-345a-4cde-80e5-c6d17e9a6ff5 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:54.885229 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292c6169-6011-4f7d-b027-e93c17b7a4cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:54.886184 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292c6169-6011-4f7d-b027-e93c17b7a4cf None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:54.886480 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292c6169-6011-4f7d-b027-e93c17b7a4cf None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:54.886766 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292c6169-6011-4f7d-b027-e93c17b7a4cf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:54.897788 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292c6169-6011-4f7d-b027-e93c17b7a4cf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:54.899258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292c6169-6011-4f7d-b027-e93c17b7a4cf None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:54.917389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-292c6169-6011-4f7d-b027-e93c17b7a4cf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:54.917389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:54.917389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:54.917389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:54.917389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:54.917389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:54.917389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:54.917389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:54.917389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:54.917389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:54.917389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:54.917389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:54.917389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:54.917389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:54.917389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:54.917389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:54.919252 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:54.919252 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:54.919252 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:54.919252 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:54.919252 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:54.919252 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:54.919252 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:54.919252 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:54.919252 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:54.919252 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:54.919252 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:54.919252 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:54.919252 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:54.919252 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:54.919252 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:54.919252 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:54.919252 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:54.926554 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:54.926554 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:54.926554 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:54.926554 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 540/1044] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:34:54.926554 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292c6169-6011-4f7d-b027-e93c17b7a4cf None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:54.941185 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f15c94-7892-4689-87ad-3a33e8593329 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:54.982870 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97928d84-ac27-4f34-bcde-380820cf0a7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:54.983474 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97928d84-ac27-4f34-bcde-380820cf0a7a None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:54.983785 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97928d84-ac27-4f34-bcde-380820cf0a7a None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:54.984059 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97928d84-ac27-4f34-bcde-380820cf0a7a None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:54.985987 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97928d84-ac27-4f34-bcde-380820cf0a7a None admin] RBAC: Adding query filter params (name=image) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:54.986308 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97928d84-ac27-4f34-bcde-380820cf0a7a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:54.987716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97928d84-ac27-4f34-bcde-380820cf0a7a None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:54.993214 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 505/1045] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:54 2022] GET /identity/v3/services?name=image => generated 123 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:54.999257 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97928d84-ac27-4f34-bcde-380820cf0a7a None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:55.055975 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292c6169-6011-4f7d-b027-e93c17b7a4cf None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:55.083502 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:55.095591 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8419c472-2a32-4c27-84f7-0a3234e0f70c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:55.096243 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8419c472-2a32-4c27-84f7-0a3234e0f70c None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:55.096592 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8419c472-2a32-4c27-84f7-0a3234e0f70c None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:55.096967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8419c472-2a32-4c27-84f7-0a3234e0f70c None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:55.097692 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8419c472-2a32-4c27-84f7-0a3234e0f70c None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:55.098142 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8419c472-2a32-4c27-84f7-0a3234e0f70c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:55.099719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8419c472-2a32-4c27-84f7-0a3234e0f70c None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:55.110580 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:55.111832 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97928d84-ac27-4f34-bcde-380820cf0a7a None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:55.115873 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 541/1046] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:54 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:55.121022 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8419c472-2a32-4c27-84f7-0a3234e0f70c None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:55.152731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a8c192-c4c9-438c-8ab6-2ea864b89dc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:55.153334 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a8c192-c4c9-438c-8ab6-2ea864b89dc1 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:55.153612 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a8c192-c4c9-438c-8ab6-2ea864b89dc1 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:55.153882 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a8c192-c4c9-438c-8ab6-2ea864b89dc1 None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:55.154992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a8c192-c4c9-438c-8ab6-2ea864b89dc1 None admin] RBAC: Adding query filter params (type=image) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:55.154992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a8c192-c4c9-438c-8ab6-2ea864b89dc1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:55.157112 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a8c192-c4c9-438c-8ab6-2ea864b89dc1 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:55.163659 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 506/1047] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:54 2022] GET /identity/v3/services?type=image => generated 352 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:55.167502 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a8c192-c4c9-438c-8ab6-2ea864b89dc1 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:55.234952 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8419c472-2a32-4c27-84f7-0a3234e0f70c None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:55.272457 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36cdd779-1e0e-4490-8a8a-0ee8d945e385 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:55.273108 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36cdd779-1e0e-4490-8a8a-0ee8d945e385 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:55.273385 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36cdd779-1e0e-4490-8a8a-0ee8d945e385 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:55.273656 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36cdd779-1e0e-4490-8a8a-0ee8d945e385 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:55.283907 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36cdd779-1e0e-4490-8a8a-0ee8d945e385 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:55.285389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36cdd779-1e0e-4490-8a8a-0ee8d945e385 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:55.289617 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a8c192-c4c9-438c-8ab6-2ea864b89dc1 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:55.296797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-36cdd779-1e0e-4490-8a8a-0ee8d945e385 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:55.296797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:55.296797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:55.296797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:55.296797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:55.296797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:55.296797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:55.296797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:55.296797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:55.296797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:55.296797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:55.296797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:55.296797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:34:55.296797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:34:55.296797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:34:55.297837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:34:55.297837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:55.297837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:55.297837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:55.297837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:55.297837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:55.297837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:55.297837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:55.297837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:55.297837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:55.297837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:55.297837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:55.297837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:55.297837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:55.297837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:55.297837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:55.297837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:55.303905 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:34:55.303905 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:34:55.303905 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:34:55.303905 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:34:55.303905 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:34:55.303905 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:34:55.303905 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:34:55.303905 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:55.303905 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 542/1048] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:34:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:55.303905 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36cdd779-1e0e-4490-8a8a-0ee8d945e385 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:55.334028 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e2f884-66fa-4d4e-9a72-9940e213e574 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:55.334630 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e2f884-66fa-4d4e-9a72-9940e213e574 None admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:55.334909 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e2f884-66fa-4d4e-9a72-9940e213e574 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:55.335175 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e2f884-66fa-4d4e-9a72-9940e213e574 None admin] PATH_INFO: `/v3/endpoints` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:55.335861 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e2f884-66fa-4d4e-9a72-9940e213e574 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:55.337320 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e2f884-66fa-4d4e-9a72-9940e213e574 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:55.347034 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:55.361847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 507/1049] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:34:55 2022] POST /identity/v3/endpoints => generated 329 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 15:34:55.375463 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:55.432138 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36cdd779-1e0e-4490-8a8a-0ee8d945e385 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:55.479312 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-162102d3-fc50-4347-8c63-e768e2ace66d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:55.479967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162102d3-fc50-4347-8c63-e768e2ace66d None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:55.480245 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162102d3-fc50-4347-8c63-e768e2ace66d None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:55.480512 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162102d3-fc50-4347-8c63-e768e2ace66d None admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:55.481226 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-162102d3-fc50-4347-8c63-e768e2ace66d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:55.481526 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-162102d3-fc50-4347-8c63-e768e2ace66d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:55.482978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-162102d3-fc50-4347-8c63-e768e2ace66d None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:55.499559 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 543/1050] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:34:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:55.506753 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e2f884-66fa-4d4e-9a72-9940e213e574 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:55.626280 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e2f884-66fa-4d4e-9a72-9940e213e574 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:55.662795 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c343fd1-5e22-486c-a0ea-9af04544e86e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:55.663392 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c343fd1-5e22-486c-a0ea-9af04544e86e None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:55.663711 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c343fd1-5e22-486c-a0ea-9af04544e86e None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:55.663960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c343fd1-5e22-486c-a0ea-9af04544e86e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:55.664693 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c343fd1-5e22-486c-a0ea-9af04544e86e None admin] RBAC: Adding query filter params (role.id=5cd803f4cb3d42f0b728362ffd2f43cd, scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5, user.id=e0d544379fe44438ad10289b45124374) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:55.665056 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c343fd1-5e22-486c-a0ea-9af04544e86e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:55.666481 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c343fd1-5e22-486c-a0ea-9af04544e86e None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:55.672704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 508/1051] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:34:55 2022] GET /identity/v3/role_assignments?user.id=e0d544379fe44438ad10289b45124374&scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5&role.id=5cd803f4cb3d42f0b728362ffd2f43cd => generated 260 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:56.479015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5ae337-297f-4e78-8138-91e730d2c10a None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:56.479537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5ae337-297f-4e78-8138-91e730d2c10a None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:56.479675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5ae337-297f-4e78-8138-91e730d2c10a None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:56.480983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 544/1052] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:56 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:56.486688 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245171ed-5891-4ab3-b99f-ef98538a4bfd None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:56.487118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245171ed-5891-4ab3-b99f-ef98538a4bfd None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:56.487352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245171ed-5891-4ab3-b99f-ef98538a4bfd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:56.506112 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-245171ed-5891-4ab3-b99f-ef98538a4bfd None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:56.507314 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-245171ed-5891-4ab3-b99f-ef98538a4bfd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:56.565763 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:56.583889 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 509/1053] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:56 2022] POST /identity/v3/auth/tokens => generated 2123 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:34:56.850611 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5ae337-297f-4e78-8138-91e730d2c10a None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:56.963149 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5ae337-297f-4e78-8138-91e730d2c10a None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:56.987026 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:57.007583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d834470e-9aaa-4354-841e-7b47bb323228 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:57.008249 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d834470e-9aaa-4354-841e-7b47bb323228 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:57.008536 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d834470e-9aaa-4354-841e-7b47bb323228 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:57.008814 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d834470e-9aaa-4354-841e-7b47bb323228 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:57.021176 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d834470e-9aaa-4354-841e-7b47bb323228 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:34:57.026183 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-d834470e-9aaa-4354-841e-7b47bb323228 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:34:57.026183 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:57.026183 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:57.026183 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:57.026183 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:57.026183 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:57.026183 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:57.026183 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:57.026183 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:57.026183 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:57.026183 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:57.026183 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:57.026183 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:34:57.026183 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:34:57.026183 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:34:57.026183 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:34:57.027102 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:34:57.027102 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:34:57.027102 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:34:57.027102 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:34:57.027102 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:57.027102 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:57.027102 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:57.027102 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:57.027102 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:57.027102 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:57.027102 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:57.027102 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:57.027102 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:57.027102 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:57.027102 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:57.027102 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:57.028516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:57.028516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:57.028516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:57.028516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:57.028516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:34:57.028516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:34:57.028516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:34:57.028516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:34:57.028516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:34:57.028516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:34:57.028516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:34:57.028516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:57.032269 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 545/1054] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:34:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:57.034108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245171ed-5891-4ab3-b99f-ef98538a4bfd None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:57.137755 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245171ed-5891-4ab3-b99f-ef98538a4bfd None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:57.175577 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41088ffb-3bb8-497c-bd23-ce6f8ad5b98a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:57.176230 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41088ffb-3bb8-497c-bd23-ce6f8ad5b98a None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:57.176586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41088ffb-3bb8-497c-bd23-ce6f8ad5b98a None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:57.176861 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41088ffb-3bb8-497c-bd23-ce6f8ad5b98a None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:57.177542 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41088ffb-3bb8-497c-bd23-ce6f8ad5b98a None admin] RBAC: Adding query filter params (name=admin) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:57.177848 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41088ffb-3bb8-497c-bd23-ce6f8ad5b98a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:57.179226 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41088ffb-3bb8-497c-bd23-ce6f8ad5b98a None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:57.188880 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 510/1055] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:34:57 2022] GET /identity/v3/roles?name=admin => generated 341 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:57.194774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d834470e-9aaa-4354-841e-7b47bb323228 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:57.299272 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d834470e-9aaa-4354-841e-7b47bb323228 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:57.333937 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c86011-8820-4804-8349-eb3b8791b14d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:57.334574 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c86011-8820-4804-8349-eb3b8791b14d None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:57.334858 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c86011-8820-4804-8349-eb3b8791b14d None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:57.335134 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c86011-8820-4804-8349-eb3b8791b14d None admin] PATH_INFO: `/v3/users/3ad7f9a6149a469ca8d9a5f3db16d657` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:57.351524 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c86011-8820-4804-8349-eb3b8791b14d None admin] RBAC: Authorizing `identity:get_user(user_id=3ad7f9a6149a469ca8d9a5f3db16d657)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:57.352794 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c86011-8820-4804-8349-eb3b8791b14d None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:57.354734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 546/1056] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:34:57 2022] GET /identity/v3/users/3ad7f9a6149a469ca8d9a5f3db16d657 => generated 286 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:57.362283 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41088ffb-3bb8-497c-bd23-ce6f8ad5b98a None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:57.422066 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:57.480690 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41088ffb-3bb8-497c-bd23-ce6f8ad5b98a None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:57.518202 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d9aae4-6f9e-484f-82b0-71033ef1ec30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:57.518950 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d9aae4-6f9e-484f-82b0-71033ef1ec30 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:57.519169 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d9aae4-6f9e-484f-82b0-71033ef1ec30 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:57.519386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d9aae4-6f9e-484f-82b0-71033ef1ec30 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:57.590682 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d9aae4-6f9e-484f-82b0-71033ef1ec30 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:57.592177 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d9aae4-6f9e-484f-82b0-71033ef1ec30 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:57.644536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:57.705084 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 511/1057] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:34:57 2022] GET /identity/v3/auth/tokens => generated 2123 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 15:34:57.706625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c86011-8820-4804-8349-eb3b8791b14d None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:57.743339 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:57.817734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c86011-8820-4804-8349-eb3b8791b14d None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:57.861946 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee5a7d0-ffd4-41e5-b89d-d500d3617155 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:57.862590 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee5a7d0-ffd4-41e5-b89d-d500d3617155 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:57.862877 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee5a7d0-ffd4-41e5-b89d-d500d3617155 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:57.863156 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee5a7d0-ffd4-41e5-b89d-d500d3617155 None admin] PATH_INFO: `/v3/projects/3e9e9ad310a14001a620aa2149e177a5` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:57.864425 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee5a7d0-ffd4-41e5-b89d-d500d3617155 None admin] RBAC: Authorizing `identity:get_project(project_id=3e9e9ad310a14001a620aa2149e177a5)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:57.865892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee5a7d0-ffd4-41e5-b89d-d500d3617155 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:57.872621 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 547/1058] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:34:57 2022] GET /identity/v3/projects/3e9e9ad310a14001a620aa2149e177a5 => generated 312 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:57.874158 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d9aae4-6f9e-484f-82b0-71033ef1ec30 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:57.930119 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015bba59-5485-4356-b5e5-dd829d362e2e None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:57.930586 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015bba59-5485-4356-b5e5-dd829d362e2e None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:57.931037 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015bba59-5485-4356-b5e5-dd829d362e2e None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:57.936129 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 548/1059] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:57 2022] GET /identity => generated 271 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:57.941397 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4649032-b859-46fa-a1c6-624b176cf6ea None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:57.941892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4649032-b859-46fa-a1c6-624b176cf6ea None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:57.942618 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4649032-b859-46fa-a1c6-624b176cf6ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:57.971919 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-d4649032-b859-46fa-a1c6-624b176cf6ea None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:57.973800 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-d4649032-b859-46fa-a1c6-624b176cf6ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:58.016501 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d9aae4-6f9e-484f-82b0-71033ef1ec30 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:58.068482 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35ccbff-e5fd-48b0-84a1-f2ca2063968d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:58.069471 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35ccbff-e5fd-48b0-84a1-f2ca2063968d None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:58.069754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35ccbff-e5fd-48b0-84a1-f2ca2063968d None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:58.070033 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35ccbff-e5fd-48b0-84a1-f2ca2063968d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:58.070773 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35ccbff-e5fd-48b0-84a1-f2ca2063968d None admin] RBAC: Adding query filter params (role.id=4d8fbbaea81a4145b89af12350bb067c, scope.project.id=3e9e9ad310a14001a620aa2149e177a5, user.id=3ad7f9a6149a469ca8d9a5f3db16d657) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:58.071089 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35ccbff-e5fd-48b0-84a1-f2ca2063968d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:58.072569 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35ccbff-e5fd-48b0-84a1-f2ca2063968d None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:58.078387 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 512/1060] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:34:57 2022] GET /identity/v3/role_assignments?user.id=3ad7f9a6149a469ca8d9a5f3db16d657&scope.project.id=3e9e9ad310a14001a620aa2149e177a5&role.id=4d8fbbaea81a4145b89af12350bb067c => generated 260 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:58.079443 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 549/1061] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:57 2022] POST /identity/v3/auth/tokens => generated 2123 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:34:58.224805 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2229c707-2ffe-45f4-8254-e885b295f29f None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:58.224987 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2229c707-2ffe-45f4-8254-e885b295f29f None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:58.225270 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2229c707-2ffe-45f4-8254-e885b295f29f None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:58.226596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 513/1062] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:34:58 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:34:58.231814 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef5912f-49e9-4e42-8f5c-1b321cf7249f None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:58.232104 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef5912f-49e9-4e42-8f5c-1b321cf7249f None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:58.232379 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef5912f-49e9-4e42-8f5c-1b321cf7249f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:58.250943 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-bef5912f-49e9-4e42-8f5c-1b321cf7249f None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:34:58.252150 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-bef5912f-49e9-4e42-8f5c-1b321cf7249f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:34:58.285070 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:58.336958 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 550/1063] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:34:58 2022] POST /identity/v3/auth/tokens => generated 2123 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:34:58.351919 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2229c707-2ffe-45f4-8254-e885b295f29f None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:58.369267 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:58.464757 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2229c707-2ffe-45f4-8254-e885b295f29f None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:58.502312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f503dff8-7acb-4499-aeb8-95769e64122d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:58.502836 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f503dff8-7acb-4499-aeb8-95769e64122d None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:58.503139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f503dff8-7acb-4499-aeb8-95769e64122d None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:58.503386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f503dff8-7acb-4499-aeb8-95769e64122d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:58.578524 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f503dff8-7acb-4499-aeb8-95769e64122d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:58.580919 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f503dff8-7acb-4499-aeb8-95769e64122d None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:58.596890 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:58.604761 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef5912f-49e9-4e42-8f5c-1b321cf7249f None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:58.693178 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 514/1064] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:34:58 2022] GET /identity/v3/auth/tokens => generated 2123 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 15:34:58.699129 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f503dff8-7acb-4499-aeb8-95769e64122d None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:58.715155 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef5912f-49e9-4e42-8f5c-1b321cf7249f None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:58.756019 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a5bb28-08d2-4848-91bf-212a95416ead None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:58.757424 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a5bb28-08d2-4848-91bf-212a95416ead None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:58.757923 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a5bb28-08d2-4848-91bf-212a95416ead None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:58.758397 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a5bb28-08d2-4848-91bf-212a95416ead None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:58.769739 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a5bb28-08d2-4848-91bf-212a95416ead None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:34:58.774713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-48a5bb28-08d2-4848-91bf-212a95416ead None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 24 15:34:58.774713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:58.774713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:58.774713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:58.774713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:58.774713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:58.774713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:58.774713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:58.774713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:58.774713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:58.774713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:58.774713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:58.774713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:34:58.774713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:34:58.774713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:34:58.774713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:34:58.775873 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:34:58.775873 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:34:58.775873 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:34:58.775873 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:34:58.775873 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:58.775873 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:58.775873 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:58.775873 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:58.775873 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:58.775873 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:58.775873 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:58.775873 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:58.775873 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:58.775873 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:58.775873 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:58.775873 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:58.776784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:58.776784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:58.776784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:58.776784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:58.776784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:34:58.776784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:34:58.776784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:34:58.776784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:34:58.776784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:34:58.776784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:34:58.776784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 24 15:34:58.776784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:58.780750 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 551/1065] 10.222.0.102 () {56 vars in 1125 bytes} [Wed Aug 24 15:34:58 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:34:58.782141 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a5bb28-08d2-4848-91bf-212a95416ead None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:58.809684 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:58.810129 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f503dff8-7acb-4499-aeb8-95769e64122d None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:58.832309 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:58.859912 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5a7578-28bb-4a2a-a2de-45e3dab00412 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:58.860596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5a7578-28bb-4a2a-a2de-45e3dab00412 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:58.860878 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5a7578-28bb-4a2a-a2de-45e3dab00412 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:58.861157 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5a7578-28bb-4a2a-a2de-45e3dab00412 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:58.871516 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f5a7578-28bb-4a2a-a2de-45e3dab00412 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:58.873021 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f5a7578-28bb-4a2a-a2de-45e3dab00412 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:58.888165 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-7f5a7578-28bb-4a2a-a2de-45e3dab00412 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:58.888165 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:58.888165 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:58.888165 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:58.888165 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:58.888165 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:58.888165 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:58.888165 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:58.888165 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:58.888165 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:58.888165 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:58.888165 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:58.888165 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:58.888165 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:58.888165 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:58.888165 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:58.889185 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:58.889185 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:58.889185 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:58.889185 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:58.889185 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:58.889185 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:58.889185 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:58.889185 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:58.889185 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:58.889185 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:58.889185 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:58.889185 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:58.889185 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:58.889185 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:58.889185 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:58.889185 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:58.889185 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:58.890113 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:58.890113 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:58.890113 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:34:58.890113 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 515/1066] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:34:58.895907 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5a7578-28bb-4a2a-a2de-45e3dab00412 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:58.900318 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a5bb28-08d2-4848-91bf-212a95416ead None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:58.939528 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc91dd71-e390-4290-ac8e-6bc9aa7aa1c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:58.940146 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc91dd71-e390-4290-ac8e-6bc9aa7aa1c5 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:58.940754 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc91dd71-e390-4290-ac8e-6bc9aa7aa1c5 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:58.940754 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc91dd71-e390-4290-ac8e-6bc9aa7aa1c5 None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:58.941358 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc91dd71-e390-4290-ac8e-6bc9aa7aa1c5 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:58.941665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc91dd71-e390-4290-ac8e-6bc9aa7aa1c5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:58.943296 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc91dd71-e390-4290-ac8e-6bc9aa7aa1c5 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:58.952450 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 552/1067] 10.222.0.102 () {56 vars in 1120 bytes} [Wed Aug 24 15:34:58 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 340 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:58.958033 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc91dd71-e390-4290-ac8e-6bc9aa7aa1c5 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:59.014361 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5a7578-28bb-4a2a-a2de-45e3dab00412 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:59.060495 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7614dcc1-0efb-4aa5-844c-c042391d87db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:59.061111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7614dcc1-0efb-4aa5-844c-c042391d87db None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:59.061401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7614dcc1-0efb-4aa5-844c-c042391d87db None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:59.061667 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7614dcc1-0efb-4aa5-844c-c042391d87db None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:59.062324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7614dcc1-0efb-4aa5-844c-c042391d87db None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:59.062628 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7614dcc1-0efb-4aa5-844c-c042391d87db None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:59.064866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7614dcc1-0efb-4aa5-844c-c042391d87db None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:59.066770 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:59.082847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 516/1068] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:58 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:34:59.096213 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc91dd71-e390-4290-ac8e-6bc9aa7aa1c5 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:59.141632 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a3e43c-0a37-4200-ac89-41bc174d14b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:59.142260 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a3e43c-0a37-4200-ac89-41bc174d14b4 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:59.142533 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a3e43c-0a37-4200-ac89-41bc174d14b4 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:59.142853 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a3e43c-0a37-4200-ac89-41bc174d14b4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:59.156014 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a3e43c-0a37-4200-ac89-41bc174d14b4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:59.157602 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a3e43c-0a37-4200-ac89-41bc174d14b4 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:59.175897 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-b7a3e43c-0a37-4200-ac89-41bc174d14b4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:59.175897 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:59.175897 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:59.175897 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:59.175897 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:59.175897 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:59.175897 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:59.175897 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:59.175897 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:59.175897 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:59.175897 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:59.175897 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:59.175897 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:34:59.175897 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:34:59.175897 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:34:59.175897 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:34:59.177192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:59.177192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:59.177192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:59.177192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:59.177192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:59.177192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:59.177192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:59.177192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:59.177192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:59.177192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:59.177192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:59.177192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:59.177192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:59.177192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:59.177192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:59.177192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:59.177192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:34:59.183821 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:34:59.183821 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:34:59.183821 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:59.183821 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 553/1069] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:34:59.184066 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7614dcc1-0efb-4aa5-844c-c042391d87db None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:59.216929 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:59.315560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7614dcc1-0efb-4aa5-844c-c042391d87db None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:59.352624 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b924306-f45d-4c0e-9f83-babab4327425 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:59.353801 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b924306-f45d-4c0e-9f83-babab4327425 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:59.354347 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b924306-f45d-4c0e-9f83-babab4327425 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:59.354643 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b924306-f45d-4c0e-9f83-babab4327425 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:59.355323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b924306-f45d-4c0e-9f83-babab4327425 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:59.355683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b924306-f45d-4c0e-9f83-babab4327425 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:59.357772 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b924306-f45d-4c0e-9f83-babab4327425 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:59.373555 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 517/1070] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:34:59 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:59.380736 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a3e43c-0a37-4200-ac89-41bc174d14b4 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:59.541063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a3e43c-0a37-4200-ac89-41bc174d14b4 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:59.593354 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d75b07d-c2cc-495e-b105-a266b11f9cfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:59.593910 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d75b07d-c2cc-495e-b105-a266b11f9cfe None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:59.594720 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d75b07d-c2cc-495e-b105-a266b11f9cfe None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:59.595025 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d75b07d-c2cc-495e-b105-a266b11f9cfe None admin] PATH_INFO: `/v3/users/nova` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:59.598274 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:59.609174 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d75b07d-c2cc-495e-b105-a266b11f9cfe None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:59.610649 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d75b07d-c2cc-495e-b105-a266b11f9cfe None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:59.626772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-3d75b07d-c2cc-495e-b105-a266b11f9cfe None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 24 15:34:59.626772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:34:59.626772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:34:59.626772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:34:59.626772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:34:59.626772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:34:59.626772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:34:59.626772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:34:59.626772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:34:59.626772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:34:59.626772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:34:59.626772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:34:59.626772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:34:59.626772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:34:59.626772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:34:59.626772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:34:59.628210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:34:59.628210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:34:59.628210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:34:59.628210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:59.628210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:34:59.628210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:34:59.628210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:34:59.628210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:34:59.628210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:34:59.628210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:34:59.628210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:34:59.628210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:34:59.628210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:34:59.628210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:34:59.628210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:34:59.628210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:34:59.628210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:34:59.628210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:34:59.629169 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:34:59.629169 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:34:59.629169 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:34:59.629169 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:34:59.629169 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:34:59.629169 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:34:59.629169 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:34:59.629169 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:34:59.629169 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 24 15:34:59.629169 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:34:59.629169 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 554/1071] 10.222.0.102 () {56 vars in 1098 bytes} [Wed Aug 24 15:34:59 2022] GET /identity/v3/users/nova => generated 82 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:34:59.636622 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b924306-f45d-4c0e-9f83-babab4327425 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:59.750496 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:34:59.772883 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b924306-f45d-4c0e-9f83-babab4327425 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:34:59.822967 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2629e1a-4991-48a1-8161-e8b41cf37ecb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:34:59.823986 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2629e1a-4991-48a1-8161-e8b41cf37ecb None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:34:59.824445 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2629e1a-4991-48a1-8161-e8b41cf37ecb None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:34:59.824933 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2629e1a-4991-48a1-8161-e8b41cf37ecb None admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:34:59.826262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2629e1a-4991-48a1-8161-e8b41cf37ecb None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:34:59.826748 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2629e1a-4991-48a1-8161-e8b41cf37ecb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:34:59.828315 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2629e1a-4991-48a1-8161-e8b41cf37ecb None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:34:59.858534 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 518/1072] 10.222.0.102 () {56 vars in 1138 bytes} [Wed Aug 24 15:34:59 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 370 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:34:59.860253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d75b07d-c2cc-495e-b105-a266b11f9cfe None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:34:59.989488 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d75b07d-c2cc-495e-b105-a266b11f9cfe None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:00.044628 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54af68e-b389-4e02-8d2b-f6c1ccbc73e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:00.045276 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54af68e-b389-4e02-8d2b-f6c1ccbc73e5 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:00.045550 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54af68e-b389-4e02-8d2b-f6c1ccbc73e5 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:00.045817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54af68e-b389-4e02-8d2b-f6c1ccbc73e5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:00.060051 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54af68e-b389-4e02-8d2b-f6c1ccbc73e5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:00.061519 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54af68e-b389-4e02-8d2b-f6c1ccbc73e5 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:00.083874 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-d54af68e-b389-4e02-8d2b-f6c1ccbc73e5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:35:00.083874 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:35:00.083874 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:35:00.083874 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:35:00.083874 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:35:00.083874 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:35:00.083874 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:35:00.083874 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:35:00.083874 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:35:00.083874 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:35:00.083874 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:35:00.083874 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:35:00.083874 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:35:00.083874 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:35:00.083874 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:35:00.083874 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:35:00.084978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:35:00.084978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:35:00.084978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:35:00.084978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:35:00.084978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:35:00.084978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:35:00.084978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:35:00.084978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:35:00.084978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:35:00.084978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:35:00.084978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:35:00.084978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:35:00.084978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:35:00.084978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:35:00.084978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:35:00.084978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:35:00.084978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:35:00.089458 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:35:00.089458 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:35:00.089458 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:35:00.089458 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 555/1073] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:34:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:35:00.089911 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef3545f-32b1-409c-bcbb-7d180c0ff486 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:00.089911 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef3545f-32b1-409c-bcbb-7d180c0ff486 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:00.089911 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef3545f-32b1-409c-bcbb-7d180c0ff486 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:00.089911 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 519/1074] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:35:00 2022] GET /identity => generated 271 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:35:00.094476 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54af68e-b389-4e02-8d2b-f6c1ccbc73e5 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:00.096076 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936d29b6-b969-4221-acf0-cc70a3ff0793 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:00.096390 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936d29b6-b969-4221-acf0-cc70a3ff0793 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:00.096677 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936d29b6-b969-4221-acf0-cc70a3ff0793 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:00.121985 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-936d29b6-b969-4221-acf0-cc70a3ff0793 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:35:00.123406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-936d29b6-b969-4221-acf0-cc70a3ff0793 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:35:00.171223 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:00.218108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 520/1075] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:35:00 2022] POST /identity/v3/auth/tokens => generated 2123 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:35:00.236977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54af68e-b389-4e02-8d2b-f6c1ccbc73e5 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:00.283059 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b78647-23c5-4fb3-8d55-c4ebcb2b972f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:00.283734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b78647-23c5-4fb3-8d55-c4ebcb2b972f None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:00.284074 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b78647-23c5-4fb3-8d55-c4ebcb2b972f None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:00.284340 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b78647-23c5-4fb3-8d55-c4ebcb2b972f None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:00.285015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b78647-23c5-4fb3-8d55-c4ebcb2b972f None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:35:00.285328 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b78647-23c5-4fb3-8d55-c4ebcb2b972f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:00.286732 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b78647-23c5-4fb3-8d55-c4ebcb2b972f None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:00.301217 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 556/1076] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:35:00 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:00.307193 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936d29b6-b969-4221-acf0-cc70a3ff0793 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:00.399186 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:00.429749 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936d29b6-b969-4221-acf0-cc70a3ff0793 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:00.483823 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d9ac5e-8723-4a25-bf18-25638066f412 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:00.484600 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d9ac5e-8723-4a25-bf18-25638066f412 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:00.485500 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d9ac5e-8723-4a25-bf18-25638066f412 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:00.485871 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d9ac5e-8723-4a25-bf18-25638066f412 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:00.498355 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d9ac5e-8723-4a25-bf18-25638066f412 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:00.499872 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d9ac5e-8723-4a25-bf18-25638066f412 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:00.513909 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-47d9ac5e-8723-4a25-bf18-25638066f412 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:35:00.513909 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:35:00.513909 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:35:00.513909 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:35:00.513909 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:35:00.513909 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:35:00.513909 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:35:00.513909 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:35:00.513909 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:35:00.513909 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:35:00.513909 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:35:00.513909 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:35:00.513909 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:35:00.513909 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:35:00.513909 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:35:00.515177 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:35:00.515177 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:35:00.515177 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:35:00.515177 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:35:00.515177 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:35:00.515177 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:35:00.515177 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:35:00.515177 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:35:00.515177 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:35:00.515177 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:35:00.515177 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:35:00.515177 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:35:00.515177 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:35:00.515177 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:35:00.515177 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:35:00.515177 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:35:00.515177 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:35:00.516746 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:35:00.516746 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:35:00.516746 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:35:00.516746 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:35:00.516746 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:35:00.516746 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:35:00.516746 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:35:00.516746 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:35:00.516746 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 521/1077] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:35:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:35:00.521739 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b78647-23c5-4fb3-8d55-c4ebcb2b972f None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:00.569442 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d9ac5e-8723-4a25-bf18-25638066f412 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:00.637665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b78647-23c5-4fb3-8d55-c4ebcb2b972f None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:00.648172 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:00.675219 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d2bb1c8-4c72-47a4-82ba-aeb4cbb58820 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:00.675888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2bb1c8-4c72-47a4-82ba-aeb4cbb58820 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:00.677533 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2bb1c8-4c72-47a4-82ba-aeb4cbb58820 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:00.677533 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2bb1c8-4c72-47a4-82ba-aeb4cbb58820 None admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:00.677533 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d2bb1c8-4c72-47a4-82ba-aeb4cbb58820 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:35:00.677533 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d2bb1c8-4c72-47a4-82ba-aeb4cbb58820 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:00.678984 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d2bb1c8-4c72-47a4-82ba-aeb4cbb58820 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:00.690181 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d9ac5e-8723-4a25-bf18-25638066f412 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:00.695761 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 557/1078] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:35:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:00.701708 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d2bb1c8-4c72-47a4-82ba-aeb4cbb58820 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:00.726280 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9ed19c-836a-429a-a25e-39a0d4540c0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:00.726909 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9ed19c-836a-429a-a25e-39a0d4540c0d None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:00.727184 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9ed19c-836a-429a-a25e-39a0d4540c0d None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:00.727461 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9ed19c-836a-429a-a25e-39a0d4540c0d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:00.729409 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:00.740545 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9ed19c-836a-429a-a25e-39a0d4540c0d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:35:00.745614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-ff9ed19c-836a-429a-a25e-39a0d4540c0d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:35:00.745614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:35:00.745614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:35:00.745614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:35:00.745614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:35:00.745614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:35:00.745614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:35:00.745614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:35:00.745614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:35:00.745614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:35:00.745614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:35:00.745614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:35:00.745614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:35:00.745614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:35:00.745614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:35:00.745614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:35:00.746700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:35:00.746700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:35:00.746700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:35:00.746700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:35:00.746700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:35:00.746700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:35:00.746700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:35:00.746700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:35:00.746700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:35:00.746700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:35:00.746700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:35:00.746700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:35:00.746700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:35:00.746700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:35:00.746700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:35:00.746700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:35:00.748333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:35:00.748333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:35:00.748333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:35:00.748333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:35:00.748333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:35:00.748333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:35:00.748333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:35:00.748333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:35:00.748333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:35:00.748333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:35:00.748333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:35:00.748333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:35:00.748333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 522/1079] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:35:00 2022] GET /identity/v3/roles/admin => generated 83 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:35:00.753740 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9ed19c-836a-429a-a25e-39a0d4540c0d None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:00.808172 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d2bb1c8-4c72-47a4-82ba-aeb4cbb58820 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:00.851551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d47268-20e7-49db-9bb0-061df343fde7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:00.852562 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d47268-20e7-49db-9bb0-061df343fde7 None admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:00.852949 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d47268-20e7-49db-9bb0-061df343fde7 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:00.853321 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d47268-20e7-49db-9bb0-061df343fde7 None admin] PATH_INFO: `/v3/projects/3e50e6f6c1f44c648337c8d37a8e94a5/users/e0d544379fe44438ad10289b45124374/roles/5cd803f4cb3d42f0b728362ffd2f43cd` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:00.857136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d47268-20e7-49db-9bb0-061df343fde7 None admin] RBAC: Authorizing `identity:create_grant(project_id=3e50e6f6c1f44c648337c8d37a8e94a5, user_id=e0d544379fe44438ad10289b45124374, role_id=5cd803f4cb3d42f0b728362ffd2f43cd)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:00.860468 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d47268-20e7-49db-9bb0-061df343fde7 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:00.876889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 558/1080] 10.222.0.102 () {58 vars in 1444 bytes} [Wed Aug 24 15:35:00 2022] PUT /identity/v3/projects/3e50e6f6c1f44c648337c8d37a8e94a5/users/e0d544379fe44438ad10289b45124374/roles/5cd803f4cb3d42f0b728362ffd2f43cd => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 15:35:00.887190 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9ed19c-836a-429a-a25e-39a0d4540c0d None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:00.928641 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a9275c-cd10-47dd-8790-423df650b6a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:00.929177 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a9275c-cd10-47dd-8790-423df650b6a7 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:00.929460 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a9275c-cd10-47dd-8790-423df650b6a7 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:00.929729 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a9275c-cd10-47dd-8790-423df650b6a7 None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:00.930430 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a9275c-cd10-47dd-8790-423df650b6a7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:35:00.930727 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a9275c-cd10-47dd-8790-423df650b6a7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:00.932235 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a9275c-cd10-47dd-8790-423df650b6a7 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:00.942764 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 523/1081] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:35:00 2022] GET /identity/v3/roles?name=admin => generated 341 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:35:00.948823 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d47268-20e7-49db-9bb0-061df343fde7 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:00.981235 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:01.064303 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d47268-20e7-49db-9bb0-061df343fde7 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:01.099587 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f43e0c-2fbe-4479-bb6f-0c69f1893fe3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:01.100255 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f43e0c-2fbe-4479-bb6f-0c69f1893fe3 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:01.100546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f43e0c-2fbe-4479-bb6f-0c69f1893fe3 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:01.100821 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f43e0c-2fbe-4479-bb6f-0c69f1893fe3 None admin] PATH_INFO: `/v3/users/3ad7f9a6149a469ca8d9a5f3db16d657` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:01.101968 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f43e0c-2fbe-4479-bb6f-0c69f1893fe3 None admin] RBAC: Authorizing `identity:get_user(user_id=3ad7f9a6149a469ca8d9a5f3db16d657)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:01.103434 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f43e0c-2fbe-4479-bb6f-0c69f1893fe3 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:01.105149 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 559/1082] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:35:00 2022] GET /identity/v3/users/3ad7f9a6149a469ca8d9a5f3db16d657 => generated 286 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:01.112203 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a9275c-cd10-47dd-8790-423df650b6a7 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:01.177685 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:01.224866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a9275c-cd10-47dd-8790-423df650b6a7 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:01.260984 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8ed4c2-c6f3-4b5e-b293-129cd9403fde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:01.261601 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8ed4c2-c6f3-4b5e-b293-129cd9403fde None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:01.261875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8ed4c2-c6f3-4b5e-b293-129cd9403fde None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:01.262142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8ed4c2-c6f3-4b5e-b293-129cd9403fde None admin] PATH_INFO: `/v3/projects/3e9e9ad310a14001a620aa2149e177a5` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:01.263244 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8ed4c2-c6f3-4b5e-b293-129cd9403fde None admin] RBAC: Authorizing `identity:get_project(project_id=3e9e9ad310a14001a620aa2149e177a5)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:01.264704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8ed4c2-c6f3-4b5e-b293-129cd9403fde None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:01.266364 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 524/1083] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:35:01 2022] GET /identity/v3/projects/3e9e9ad310a14001a620aa2149e177a5 => generated 312 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:35:01.273429 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f43e0c-2fbe-4479-bb6f-0c69f1893fe3 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:01.439776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:01.448288 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f43e0c-2fbe-4479-bb6f-0c69f1893fe3 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:01.486029 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0aa36c5-3fec-4853-ace1-5e8716643e77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:01.486623 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0aa36c5-3fec-4853-ace1-5e8716643e77 None admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:01.486894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0aa36c5-3fec-4853-ace1-5e8716643e77 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:01.487169 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0aa36c5-3fec-4853-ace1-5e8716643e77 None admin] PATH_INFO: `/v3/projects/3e9e9ad310a14001a620aa2149e177a5/users/3ad7f9a6149a469ca8d9a5f3db16d657/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:01.489191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0aa36c5-3fec-4853-ace1-5e8716643e77 None admin] RBAC: Authorizing `identity:create_grant(project_id=3e9e9ad310a14001a620aa2149e177a5, user_id=3ad7f9a6149a469ca8d9a5f3db16d657, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:01.490639 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0aa36c5-3fec-4853-ace1-5e8716643e77 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:01.505324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 560/1084] 10.222.0.102 () {58 vars in 1444 bytes} [Wed Aug 24 15:35:01 2022] PUT /identity/v3/projects/3e9e9ad310a14001a620aa2149e177a5/users/3ad7f9a6149a469ca8d9a5f3db16d657/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:35:02.872294 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0ff957-e275-4f7e-bebb-36a1645ccf44 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:02.872820 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0ff957-e275-4f7e-bebb-36a1645ccf44 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:02.872820 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0ff957-e275-4f7e-bebb-36a1645ccf44 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:02.874185 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 525/1085] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:35:02 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:35:02.880723 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d690a39-a5a5-414d-aae4-c9d1092be02b None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:02.881310 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d690a39-a5a5-414d-aae4-c9d1092be02b None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:02.881472 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d690a39-a5a5-414d-aae4-c9d1092be02b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:02.901006 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-4d690a39-a5a5-414d-aae4-c9d1092be02b None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:35:02.902228 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-4d690a39-a5a5-414d-aae4-c9d1092be02b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:35:02.983093 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 561/1086] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:35:02 2022] POST /identity/v3/auth/tokens => generated 2123 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:35:03.249105 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0ff957-e275-4f7e-bebb-36a1645ccf44 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:03.357131 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0ff957-e275-4f7e-bebb-36a1645ccf44 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:03.382825 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:03.394546 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c2928f-b3f7-4090-83b8-98ba97731763 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:03.395178 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c2928f-b3f7-4090-83b8-98ba97731763 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:03.395457 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c2928f-b3f7-4090-83b8-98ba97731763 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:03.395765 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c2928f-b3f7-4090-83b8-98ba97731763 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:03.405976 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c2928f-b3f7-4090-83b8-98ba97731763 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:35:03.410724 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-88c2928f-b3f7-4090-83b8-98ba97731763 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 24 15:35:03.410724 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:35:03.410724 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:35:03.410724 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:35:03.410724 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:35:03.410724 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:35:03.410724 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:35:03.410724 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:35:03.410724 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:35:03.410724 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:35:03.410724 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:35:03.410724 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:35:03.410724 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:35:03.410724 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:35:03.410724 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:35:03.410724 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:35:03.411886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:35:03.411886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:35:03.411886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:35:03.411886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:35:03.411886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:35:03.411886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:35:03.411886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:35:03.411886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:35:03.411886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:35:03.411886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:35:03.411886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:35:03.411886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:35:03.411886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:35:03.411886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:35:03.411886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:35:03.411886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:35:03.412714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:35:03.412714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:35:03.412714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:35:03.412714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:35:03.412714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:35:03.412714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:35:03.412714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:35:03.412714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:35:03.412714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:35:03.412714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:35:03.412714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 24 15:35:03.412714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:35:03.413359 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93796ea3-2e15-41fc-a792-dffb9b77f432 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:03.413359 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93796ea3-2e15-41fc-a792-dffb9b77f432 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:03.413359 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93796ea3-2e15-41fc-a792-dffb9b77f432 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:03.413544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 526/1087] 10.222.0.102 () {56 vars in 1125 bytes} [Wed Aug 24 15:35:03 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:35:03.416816 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 562/1088] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:35:03 2022] GET /identity => generated 271 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:35:03.417556 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c2928f-b3f7-4090-83b8-98ba97731763 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:03.421892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c6b42c-9709-4f92-8bd3-fc303700080e None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:03.422178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c6b42c-9709-4f92-8bd3-fc303700080e None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:03.422987 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c6b42c-9709-4f92-8bd3-fc303700080e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:03.440981 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-d6c6b42c-9709-4f92-8bd3-fc303700080e None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:35:03.442168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-d6c6b42c-9709-4f92-8bd3-fc303700080e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:35:03.512883 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 563/1089] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:35:03 2022] POST /identity/v3/auth/tokens => generated 2123 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 15:35:03.523876 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c2928f-b3f7-4090-83b8-98ba97731763 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:03.560438 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94876d6-a965-4d5c-835b-bf791e13f2e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:03.561045 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94876d6-a965-4d5c-835b-bf791e13f2e0 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:03.561329 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94876d6-a965-4d5c-835b-bf791e13f2e0 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:03.561603 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94876d6-a965-4d5c-835b-bf791e13f2e0 None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:03.562277 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94876d6-a965-4d5c-835b-bf791e13f2e0 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:35:03.562585 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94876d6-a965-4d5c-835b-bf791e13f2e0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:03.564734 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94876d6-a965-4d5c-835b-bf791e13f2e0 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:03.573418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 527/1090] 10.222.0.102 () {56 vars in 1120 bytes} [Wed Aug 24 15:35:03 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 340 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:35:03.580165 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c6b42c-9709-4f92-8bd3-fc303700080e None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:03.598871 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:03.680731 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c6b42c-9709-4f92-8bd3-fc303700080e None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:03.717090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec30853-c405-4e3f-8a8a-4862856cb09b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:03.717714 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec30853-c405-4e3f-8a8a-4862856cb09b None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:03.717987 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec30853-c405-4e3f-8a8a-4862856cb09b None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:03.718260 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec30853-c405-4e3f-8a8a-4862856cb09b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:03.729728 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec30853-c405-4e3f-8a8a-4862856cb09b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:03.731182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec30853-c405-4e3f-8a8a-4862856cb09b None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:03.747787 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-7ec30853-c405-4e3f-8a8a-4862856cb09b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:35:03.747787 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:35:03.747787 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:35:03.747787 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:35:03.747787 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:35:03.747787 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:35:03.747787 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:35:03.747787 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:35:03.747787 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:35:03.747787 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:35:03.747787 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:35:03.747787 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:35:03.747787 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:35:03.747787 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:35:03.747787 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:35:03.747787 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:35:03.749469 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:35:03.749469 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:35:03.749469 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:35:03.749469 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:35:03.749469 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:35:03.749469 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:35:03.749469 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:35:03.749469 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:35:03.749469 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:35:03.749469 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:35:03.749469 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:35:03.749469 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:35:03.749469 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:35:03.749469 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:35:03.749469 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:35:03.749469 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:35:03.749469 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:35:03.751688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:35:03.751688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:35:03.751688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:35:03.751688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 564/1091] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:35:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:35:03.756889 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94876d6-a965-4d5c-835b-bf791e13f2e0 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:03.784039 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec30853-c405-4e3f-8a8a-4862856cb09b None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:03.816681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:03.877817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:03.885704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94876d6-a965-4d5c-835b-bf791e13f2e0 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:03.921245 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec30853-c405-4e3f-8a8a-4862856cb09b None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:03.927868 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a517f67-6826-4f30-a3bf-759e0ef39d14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:03.928483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a517f67-6826-4f30-a3bf-759e0ef39d14 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:03.928765 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a517f67-6826-4f30-a3bf-759e0ef39d14 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:03.929039 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a517f67-6826-4f30-a3bf-759e0ef39d14 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:03.929700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a517f67-6826-4f30-a3bf-759e0ef39d14 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:35:03.930000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a517f67-6826-4f30-a3bf-759e0ef39d14 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:03.931400 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a517f67-6826-4f30-a3bf-759e0ef39d14 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:03.945956 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 528/1092] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:35:03 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:35:03.952486 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a517f67-6826-4f30-a3bf-759e0ef39d14 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:03.971131 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c15451-6d6c-448a-8107-9c58b71afbe7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:03.972164 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c15451-6d6c-448a-8107-9c58b71afbe7 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:03.972665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c15451-6d6c-448a-8107-9c58b71afbe7 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:03.973024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c15451-6d6c-448a-8107-9c58b71afbe7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:03.989181 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c15451-6d6c-448a-8107-9c58b71afbe7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:35:03.995913 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-45c15451-6d6c-448a-8107-9c58b71afbe7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:35:03.995913 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:35:03.995913 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:35:03.995913 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:35:03.995913 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:35:03.995913 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:35:03.995913 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:35:03.995913 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:35:03.995913 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:35:03.995913 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:35:03.995913 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:35:03.995913 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:35:03.995913 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:35:03.995913 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:35:03.995913 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:35:03.995913 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:35:03.996892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:35:03.996892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:35:03.996892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:35:03.996892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:35:03.996892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:35:03.996892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:35:03.996892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:35:03.996892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:35:03.996892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:35:03.996892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:35:03.996892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:35:03.996892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:35:03.996892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:35:03.996892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:35:03.996892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:35:03.996892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:35:03.997970 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:35:03.997970 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:35:03.997970 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:35:03.997970 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:35:03.997970 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:35:03.997970 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:35:03.997970 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:35:03.997970 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:35:03.997970 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:35:03.997970 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:35:03.997970 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:35:03.997970 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:35:03.997970 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 565/1093] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:35:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:35:04.004602 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c15451-6d6c-448a-8107-9c58b71afbe7 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:04.073878 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a517f67-6826-4f30-a3bf-759e0ef39d14 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:04.082351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:04.113007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c15451-6d6c-448a-8107-9c58b71afbe7 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:04.113291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8105a4f4-212e-454a-a537-9866ee7ff3d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:04.113891 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8105a4f4-212e-454a-a537-9866ee7ff3d0 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:04.114167 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8105a4f4-212e-454a-a537-9866ee7ff3d0 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:04.114436 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8105a4f4-212e-454a-a537-9866ee7ff3d0 None admin] PATH_INFO: `/v3/users/nova` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:04.121798 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8105a4f4-212e-454a-a537-9866ee7ff3d0 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:04.123244 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8105a4f4-212e-454a-a537-9866ee7ff3d0 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:04.137330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-8105a4f4-212e-454a-a537-9866ee7ff3d0 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 24 15:35:04.137330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:35:04.137330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:35:04.137330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:35:04.137330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:35:04.137330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:35:04.137330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:35:04.137330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:35:04.137330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:35:04.137330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:35:04.137330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:35:04.137330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:35:04.137330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:35:04.137330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:35:04.137330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:35:04.137330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:35:04.138351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:35:04.138351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:35:04.138351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:35:04.138351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:35:04.138351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:35:04.138351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:35:04.138351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:35:04.138351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:35:04.138351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:35:04.138351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:35:04.138351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:35:04.138351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:35:04.138351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:35:04.138351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:35:04.138351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:35:04.138351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:35:04.138351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:35:04.138351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:35:04.139323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:35:04.139323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:35:04.139323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:35:04.139323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:35:04.139323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:35:04.139323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:35:04.139323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:35:04.139323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:35:04.139323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 24 15:35:04.139323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:35:04.139323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 529/1094] 10.222.0.102 () {56 vars in 1098 bytes} [Wed Aug 24 15:35:03 2022] GET /identity/v3/users/nova => generated 82 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:35:04.141672 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:04.144623 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8105a4f4-212e-454a-a537-9866ee7ff3d0 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:04.152116 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0466c256-4261-4e2f-a920-21cb514d9a76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:04.152731 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0466c256-4261-4e2f-a920-21cb514d9a76 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:04.153012 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0466c256-4261-4e2f-a920-21cb514d9a76 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:04.153285 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0466c256-4261-4e2f-a920-21cb514d9a76 None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:04.153961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0466c256-4261-4e2f-a920-21cb514d9a76 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:35:04.154269 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0466c256-4261-4e2f-a920-21cb514d9a76 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:04.156167 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0466c256-4261-4e2f-a920-21cb514d9a76 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:04.166093 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 566/1095] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:35:04 2022] GET /identity/v3/roles?name=admin => generated 341 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:35:04.171936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0466c256-4261-4e2f-a920-21cb514d9a76 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:04.248897 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8105a4f4-212e-454a-a537-9866ee7ff3d0 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:04.276608 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0466c256-4261-4e2f-a920-21cb514d9a76 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:04.283588 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b99951a-0f4a-4d6b-8d07-1eb875ce05d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:04.284289 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b99951a-0f4a-4d6b-8d07-1eb875ce05d0 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:04.284574 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b99951a-0f4a-4d6b-8d07-1eb875ce05d0 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:04.284847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b99951a-0f4a-4d6b-8d07-1eb875ce05d0 None admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:04.285590 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b99951a-0f4a-4d6b-8d07-1eb875ce05d0 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:35:04.285896 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b99951a-0f4a-4d6b-8d07-1eb875ce05d0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:04.287328 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b99951a-0f4a-4d6b-8d07-1eb875ce05d0 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:04.305359 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 530/1096] 10.222.0.102 () {56 vars in 1138 bytes} [Wed Aug 24 15:35:04 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 370 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:35:04.311039 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b99951a-0f4a-4d6b-8d07-1eb875ce05d0 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:04.311802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f740931a-e63f-4c66-a3de-38d3352bb867 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:04.312430 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f740931a-e63f-4c66-a3de-38d3352bb867 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:04.312709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f740931a-e63f-4c66-a3de-38d3352bb867 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:04.312977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f740931a-e63f-4c66-a3de-38d3352bb867 None admin] PATH_INFO: `/v3/users/3ad7f9a6149a469ca8d9a5f3db16d657` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:04.314782 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f740931a-e63f-4c66-a3de-38d3352bb867 None admin] RBAC: Authorizing `identity:get_user(user_id=3ad7f9a6149a469ca8d9a5f3db16d657)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:04.316250 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f740931a-e63f-4c66-a3de-38d3352bb867 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:04.318105 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 567/1097] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:35:04 2022] GET /identity/v3/users/3ad7f9a6149a469ca8d9a5f3db16d657 => generated 286 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:04.324119 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f740931a-e63f-4c66-a3de-38d3352bb867 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:04.339914 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:04.385284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:04.417123 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b99951a-0f4a-4d6b-8d07-1eb875ce05d0 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:04.428593 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f740931a-e63f-4c66-a3de-38d3352bb867 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:04.454443 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5e9176-9c14-4c9e-b4f1-369040f8dcdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:04.455067 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5e9176-9c14-4c9e-b4f1-369040f8dcdf None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:04.455344 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5e9176-9c14-4c9e-b4f1-369040f8dcdf None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:04.455613 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5e9176-9c14-4c9e-b4f1-369040f8dcdf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:04.462923 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bfd499d-b154-4101-9809-c140e3240d6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:04.463581 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfd499d-b154-4101-9809-c140e3240d6c None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:04.463904 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfd499d-b154-4101-9809-c140e3240d6c None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:04.464306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfd499d-b154-4101-9809-c140e3240d6c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:04.532329 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b5e9176-9c14-4c9e-b4f1-369040f8dcdf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:04.532735 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bfd499d-b154-4101-9809-c140e3240d6c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:04.533840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b5e9176-9c14-4c9e-b4f1-369040f8dcdf None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:04.533951 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bfd499d-b154-4101-9809-c140e3240d6c None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:04.557806 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:04.594915 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:04.634708 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 531/1098] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:35:04 2022] GET /identity/v3/auth/tokens => generated 2123 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 24 15:35:04.638083 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 568/1099] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:35:04 2022] GET /identity/v3/auth/tokens => generated 2123 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 24 15:35:04.640489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5e9176-9c14-4c9e-b4f1-369040f8dcdf None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:04.644631 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bfd499d-b154-4101-9809-c140e3240d6c None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:04.745651 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5e9176-9c14-4c9e-b4f1-369040f8dcdf None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:04.746316 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bfd499d-b154-4101-9809-c140e3240d6c None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:04.770767 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:04.781792 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc541a9-59b6-4038-9fd4-daa079dea059 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:04.782154 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f016227b-3daa-437b-937b-e6367c086a33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:04.782426 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc541a9-59b6-4038-9fd4-daa079dea059 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:04.782704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc541a9-59b6-4038-9fd4-daa079dea059 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:04.782799 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f016227b-3daa-437b-937b-e6367c086a33 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:04.782974 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc541a9-59b6-4038-9fd4-daa079dea059 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:04.783067 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f016227b-3daa-437b-937b-e6367c086a33 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:04.783315 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f016227b-3daa-437b-937b-e6367c086a33 None admin] PATH_INFO: `/v3/projects/3e9e9ad310a14001a620aa2149e177a5` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:04.784707 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f016227b-3daa-437b-937b-e6367c086a33 None admin] RBAC: Authorizing `identity:get_project(project_id=3e9e9ad310a14001a620aa2149e177a5)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:04.786138 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f016227b-3daa-437b-937b-e6367c086a33 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:04.788776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 569/1100] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:35:04 2022] GET /identity/v3/projects/3e9e9ad310a14001a620aa2149e177a5 => generated 312 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:35:04.794594 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc541a9-59b6-4038-9fd4-daa079dea059 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:04.795053 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f016227b-3daa-437b-937b-e6367c086a33 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:04.796150 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc541a9-59b6-4038-9fd4-daa079dea059 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:04.811323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-0fc541a9-59b6-4038-9fd4-daa079dea059 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:35:04.811323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:35:04.811323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:35:04.811323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:35:04.811323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:35:04.811323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:35:04.811323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:35:04.811323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:35:04.811323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:35:04.811323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:35:04.811323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:35:04.811323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:35:04.811323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:35:04.811323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:35:04.811323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:35:04.811323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:35:04.812434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:35:04.812434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:35:04.812434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:35:04.812434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:35:04.812434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:35:04.812434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:35:04.812434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:35:04.812434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:35:04.812434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:35:04.812434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:35:04.812434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:35:04.812434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:35:04.812434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:35:04.812434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:35:04.812434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:35:04.812434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:35:04.812434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:35:04.814168 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:35:04.814168 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:35:04.814168 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:35:04.814168 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 532/1101] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:35:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:35:04.818865 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc541a9-59b6-4038-9fd4-daa079dea059 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:04.828334 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:04.920472 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f016227b-3daa-437b-937b-e6367c086a33 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:04.935395 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc541a9-59b6-4038-9fd4-daa079dea059 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:04.959922 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66619264-2741-44ee-981e-118d7213499a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:04.960522 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66619264-2741-44ee-981e-118d7213499a None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:04.960800 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66619264-2741-44ee-981e-118d7213499a None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:04.961071 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66619264-2741-44ee-981e-118d7213499a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:04.961796 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66619264-2741-44ee-981e-118d7213499a None admin] RBAC: Adding query filter params (role.id=4d8fbbaea81a4145b89af12350bb067c, scope.project.id=3e9e9ad310a14001a620aa2149e177a5, user.id=3ad7f9a6149a469ca8d9a5f3db16d657) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:35:04.962111 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66619264-2741-44ee-981e-118d7213499a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:04.963537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66619264-2741-44ee-981e-118d7213499a None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:04.987663 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 570/1102] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:35:04 2022] GET /identity/v3/role_assignments?user.id=3ad7f9a6149a469ca8d9a5f3db16d657&scope.project.id=3e9e9ad310a14001a620aa2149e177a5&role.id=4d8fbbaea81a4145b89af12350bb067c => generated 610 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:35:04.987890 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c97bfd-8913-4129-b893-627ade6802f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:04.987890 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c97bfd-8913-4129-b893-627ade6802f8 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:04.987890 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c97bfd-8913-4129-b893-627ade6802f8 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:04.987890 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c97bfd-8913-4129-b893-627ade6802f8 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:04.988220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c97bfd-8913-4129-b893-627ade6802f8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:35:04.988220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c97bfd-8913-4129-b893-627ade6802f8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:04.988220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c97bfd-8913-4129-b893-627ade6802f8 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:04.992770 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 533/1103] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:35:04 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:35:04.998533 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66619264-2741-44ee-981e-118d7213499a None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:05.100177 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:05.110982 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66619264-2741-44ee-981e-118d7213499a None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:05.147377 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0304c7be-6bc6-4867-a008-2fbd91d1b290 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:05.148069 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0304c7be-6bc6-4867-a008-2fbd91d1b290 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:05.148356 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0304c7be-6bc6-4867-a008-2fbd91d1b290 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:05.148638 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0304c7be-6bc6-4867-a008-2fbd91d1b290 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:05.155553 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0304c7be-6bc6-4867-a008-2fbd91d1b290 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:05.157095 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0304c7be-6bc6-4867-a008-2fbd91d1b290 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:05.169069 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-0304c7be-6bc6-4867-a008-2fbd91d1b290 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:35:05.169069 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:35:05.169069 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:35:05.169069 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:35:05.169069 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:35:05.169069 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:35:05.169069 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:35:05.169069 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:35:05.169069 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:35:05.169069 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:35:05.169069 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:35:05.169069 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:35:05.169069 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:35:05.169069 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:35:05.169069 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:35:05.170258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:35:05.170258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:35:05.170258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:35:05.170258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:35:05.170258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:35:05.170258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:35:05.170258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:35:05.170258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:35:05.170258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:35:05.170258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:35:05.170258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:35:05.170258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:35:05.170258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:35:05.170258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:35:05.170258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:35:05.170258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:35:05.170258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:35:05.171306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:35:05.171306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:35:05.171306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:35:05.171306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:35:05.171306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:35:05.171306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:35:05.171306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:35:05.171306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:35:05.171306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 571/1104] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:35:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:35:05.176928 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c97bfd-8913-4129-b893-627ade6802f8 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:05.213231 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:05.283326 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c97bfd-8913-4129-b893-627ade6802f8 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:05.319190 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca691cc8-d7d5-4293-abb9-7d2126b9f2ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:05.319834 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca691cc8-d7d5-4293-abb9-7d2126b9f2ca None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:05.320120 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca691cc8-d7d5-4293-abb9-7d2126b9f2ca None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:05.320396 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca691cc8-d7d5-4293-abb9-7d2126b9f2ca None admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:05.321094 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca691cc8-d7d5-4293-abb9-7d2126b9f2ca None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:35:05.321403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca691cc8-d7d5-4293-abb9-7d2126b9f2ca None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:05.322833 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca691cc8-d7d5-4293-abb9-7d2126b9f2ca None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:05.336003 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 534/1105] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:35:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:05.342031 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0304c7be-6bc6-4867-a008-2fbd91d1b290 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:05.445913 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0304c7be-6bc6-4867-a008-2fbd91d1b290 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:05.481206 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ececc9-2455-4ef2-bc1e-21793369775a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:05.481819 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ececc9-2455-4ef2-bc1e-21793369775a None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:05.482101 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ececc9-2455-4ef2-bc1e-21793369775a None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:05.482375 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ececc9-2455-4ef2-bc1e-21793369775a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:05.483110 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ececc9-2455-4ef2-bc1e-21793369775a None admin] RBAC: Adding query filter params (role.id=5cd803f4cb3d42f0b728362ffd2f43cd, scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5, user.id=e0d544379fe44438ad10289b45124374) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:35:05.483426 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ececc9-2455-4ef2-bc1e-21793369775a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:05.484897 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ececc9-2455-4ef2-bc1e-21793369775a None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:05.492114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 572/1106] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:35:05 2022] GET /identity/v3/role_assignments?user.id=e0d544379fe44438ad10289b45124374&scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5&role.id=5cd803f4cb3d42f0b728362ffd2f43cd => generated 610 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:06.931856 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa006389-73d7-4bdd-b47c-4ed3f91ad2ff None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:06.932379 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa006389-73d7-4bdd-b47c-4ed3f91ad2ff None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:06.932496 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa006389-73d7-4bdd-b47c-4ed3f91ad2ff None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:06.933733 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 535/1107] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:35:06 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:35:06.939860 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344e499c-702d-407f-8a02-94062cd1b04e None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:06.940353 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344e499c-702d-407f-8a02-94062cd1b04e None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:06.940503 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344e499c-702d-407f-8a02-94062cd1b04e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:06.959144 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-344e499c-702d-407f-8a02-94062cd1b04e None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:35:06.960368 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-344e499c-702d-407f-8a02-94062cd1b04e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:35:06.969867 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:07.029028 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 573/1108] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:35:06 2022] POST /identity/v3/auth/tokens => generated 2123 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:35:07.310617 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa006389-73d7-4bdd-b47c-4ed3f91ad2ff None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:07.404307 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:07.411657 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa006389-73d7-4bdd-b47c-4ed3f91ad2ff None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:07.446984 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e83ca92c-131e-4381-9152-145ac6c0c1a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:07.447618 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83ca92c-131e-4381-9152-145ac6c0c1a9 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:07.447943 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83ca92c-131e-4381-9152-145ac6c0c1a9 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:07.448463 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83ca92c-131e-4381-9152-145ac6c0c1a9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:07.449691 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e83ca92c-131e-4381-9152-145ac6c0c1a9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:07.451120 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e83ca92c-131e-4381-9152-145ac6c0c1a9 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:07.452730 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 536/1109] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:35:07 2022] GET /identity/v3/domains/default => generated 200 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:07.458499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344e499c-702d-407f-8a02-94062cd1b04e None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:07.559921 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344e499c-702d-407f-8a02-94062cd1b04e None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:07.594414 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3541f0f2-70e5-449f-9ee6-364271f7c6f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:07.595529 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3541f0f2-70e5-449f-9ee6-364271f7c6f0 None admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:07.595831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3541f0f2-70e5-449f-9ee6-364271f7c6f0 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:07.596105 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3541f0f2-70e5-449f-9ee6-364271f7c6f0 None admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:07.596767 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3541f0f2-70e5-449f-9ee6-364271f7c6f0 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:07.598210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3541f0f2-70e5-449f-9ee6-364271f7c6f0 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:07.616767 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:07.622597 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 574/1110] 10.222.0.102 () {60 vars in 1137 bytes} [Wed Aug 24 15:35:07 2022] POST /identity/v3/users => generated 287 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:35:09.618848 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed594bb-252f-46ee-9205-f4f17bbe3c3d None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:09.619428 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed594bb-252f-46ee-9205-f4f17bbe3c3d None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:09.619428 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed594bb-252f-46ee-9205-f4f17bbe3c3d None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:09.620757 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 537/1111] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:35:09 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:35:09.626141 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25773636-d214-407a-9b84-15b8b582b21f None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:09.626582 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25773636-d214-407a-9b84-15b8b582b21f None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:09.626709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25773636-d214-407a-9b84-15b8b582b21f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:09.645744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-25773636-d214-407a-9b84-15b8b582b21f None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:35:09.646938 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-25773636-d214-407a-9b84-15b8b582b21f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:35:09.713635 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 575/1112] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:35:09 2022] POST /identity/v3/auth/tokens => generated 2123 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:35:09.993015 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed594bb-252f-46ee-9205-f4f17bbe3c3d None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:10.101587 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed594bb-252f-46ee-9205-f4f17bbe3c3d None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:10.140402 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7dd3b3b-72d3-47db-a901-d0122ca66452 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:10.141027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dd3b3b-72d3-47db-a901-d0122ca66452 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:10.141307 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dd3b3b-72d3-47db-a901-d0122ca66452 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:10.141913 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dd3b3b-72d3-47db-a901-d0122ca66452 None admin] PATH_INFO: `/v3/roles/b8429ebd9c2a4cbbbbf30ad56ce23bfa` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:10.143313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7dd3b3b-72d3-47db-a901-d0122ca66452 None admin] RBAC: Authorizing `identity:get_role(role_id=b8429ebd9c2a4cbbbbf30ad56ce23bfa)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:10.144726 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7dd3b3b-72d3-47db-a901-d0122ca66452 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:10.146622 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 538/1113] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:35:09 2022] GET /identity/v3/roles/b8429ebd9c2a4cbbbbf30ad56ce23bfa => generated 224 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:10.152572 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25773636-d214-407a-9b84-15b8b582b21f None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:10.264930 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25773636-d214-407a-9b84-15b8b582b21f None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:10.297267 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:10.304110 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7464068d-4da6-4c49-bdca-05b494b3e734 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:10.304747 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7464068d-4da6-4c49-bdca-05b494b3e734 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:10.305030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7464068d-4da6-4c49-bdca-05b494b3e734 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:10.305307 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7464068d-4da6-4c49-bdca-05b494b3e734 None admin] PATH_INFO: `/v3/users/300bbe096fcf4fabb645fbe2fbc075d4` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:10.320176 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7464068d-4da6-4c49-bdca-05b494b3e734 None admin] RBAC: Authorizing `identity:get_user(user_id=300bbe096fcf4fabb645fbe2fbc075d4)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:10.321618 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7464068d-4da6-4c49-bdca-05b494b3e734 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:10.323239 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 576/1114] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:35:10 2022] GET /identity/v3/users/300bbe096fcf4fabb645fbe2fbc075d4 => generated 287 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:10.329519 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7dd3b3b-72d3-47db-a901-d0122ca66452 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:10.361720 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:10.430842 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7dd3b3b-72d3-47db-a901-d0122ca66452 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:10.463679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ebecc45-58ff-4d85-9307-a42362d04801 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:10.464274 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebecc45-58ff-4d85-9307-a42362d04801 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:10.464547 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebecc45-58ff-4d85-9307-a42362d04801 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:10.464813 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebecc45-58ff-4d85-9307-a42362d04801 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:10.530578 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ebecc45-58ff-4d85-9307-a42362d04801 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:10.532018 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ebecc45-58ff-4d85-9307-a42362d04801 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:10.563855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:10.632828 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 539/1115] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:35:10 2022] GET /identity/v3/auth/tokens => generated 2123 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 15:35:10.638904 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7464068d-4da6-4c49-bdca-05b494b3e734 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:10.752556 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7464068d-4da6-4c49-bdca-05b494b3e734 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:10.787482 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fa030c-ad66-4305-8d80-6852e079a224 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:10.788067 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fa030c-ad66-4305-8d80-6852e079a224 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:10.788351 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fa030c-ad66-4305-8d80-6852e079a224 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:10.788626 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fa030c-ad66-4305-8d80-6852e079a224 None admin] PATH_INFO: `/v3/projects/3e9e9ad310a14001a620aa2149e177a5` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:10.789744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72fa030c-ad66-4305-8d80-6852e079a224 None admin] RBAC: Authorizing `identity:get_project(project_id=3e9e9ad310a14001a620aa2149e177a5)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:10.791170 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72fa030c-ad66-4305-8d80-6852e079a224 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:10.793038 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 577/1116] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:35:10 2022] GET /identity/v3/projects/3e9e9ad310a14001a620aa2149e177a5 => generated 312 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:10.800114 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ebecc45-58ff-4d85-9307-a42362d04801 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:10.900245 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ebecc45-58ff-4d85-9307-a42362d04801 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:10.931037 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:10.936591 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6318ba-fbff-4b39-92b2-fdd497eb403c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:10.937191 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6318ba-fbff-4b39-92b2-fdd497eb403c None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:10.937468 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6318ba-fbff-4b39-92b2-fdd497eb403c None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:10.937738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6318ba-fbff-4b39-92b2-fdd497eb403c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:10.938621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab6318ba-fbff-4b39-92b2-fdd497eb403c None admin] RBAC: Adding query filter params (role.id=b8429ebd9c2a4cbbbbf30ad56ce23bfa, scope.project.id=3e9e9ad310a14001a620aa2149e177a5, user.id=300bbe096fcf4fabb645fbe2fbc075d4) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:35:10.940119 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab6318ba-fbff-4b39-92b2-fdd497eb403c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:10.941564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab6318ba-fbff-4b39-92b2-fdd497eb403c None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:10.947354 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 540/1117] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:35:10 2022] GET /identity/v3/role_assignments?user.id=300bbe096fcf4fabb645fbe2fbc075d4&scope.project.id=3e9e9ad310a14001a620aa2149e177a5&role.id=b8429ebd9c2a4cbbbbf30ad56ce23bfa => generated 260 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:12.860218 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba0184e-70b8-4d03-9b38-e751643eb0c8 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:12.860724 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba0184e-70b8-4d03-9b38-e751643eb0c8 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:12.860824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba0184e-70b8-4d03-9b38-e751643eb0c8 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:12.862270 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 578/1118] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:35:12 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:35:12.867433 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31201ad6-e43e-485c-8fea-81cf6f95fe5f None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:12.867983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31201ad6-e43e-485c-8fea-81cf6f95fe5f None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:12.867983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31201ad6-e43e-485c-8fea-81cf6f95fe5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:12.887707 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-31201ad6-e43e-485c-8fea-81cf6f95fe5f None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:35:12.888923 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-31201ad6-e43e-485c-8fea-81cf6f95fe5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:35:12.963404 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 541/1119] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:35:12 2022] POST /identity/v3/auth/tokens => generated 2123 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:35:13.232036 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba0184e-70b8-4d03-9b38-e751643eb0c8 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:13.292214 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:13.338246 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba0184e-70b8-4d03-9b38-e751643eb0c8 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:13.371736 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99bc76d4-1601-48e2-a2f8-a1769fb0b1a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:13.372356 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bc76d4-1601-48e2-a2f8-a1769fb0b1a1 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:13.372634 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bc76d4-1601-48e2-a2f8-a1769fb0b1a1 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:13.372904 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bc76d4-1601-48e2-a2f8-a1769fb0b1a1 None admin] PATH_INFO: `/v3/roles/b8429ebd9c2a4cbbbbf30ad56ce23bfa` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:13.374318 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99bc76d4-1601-48e2-a2f8-a1769fb0b1a1 None admin] RBAC: Authorizing `identity:get_role(role_id=b8429ebd9c2a4cbbbbf30ad56ce23bfa)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:13.375701 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99bc76d4-1601-48e2-a2f8-a1769fb0b1a1 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:13.377693 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 579/1120] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:35:13 2022] GET /identity/v3/roles/b8429ebd9c2a4cbbbbf30ad56ce23bfa => generated 224 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:13.383708 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31201ad6-e43e-485c-8fea-81cf6f95fe5f None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:13.483747 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31201ad6-e43e-485c-8fea-81cf6f95fe5f None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:13.510006 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:13.519661 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bcaadb7-9c32-441e-9ddf-5015dce59330 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:13.520297 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcaadb7-9c32-441e-9ddf-5015dce59330 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:13.520580 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcaadb7-9c32-441e-9ddf-5015dce59330 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:13.520859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcaadb7-9c32-441e-9ddf-5015dce59330 None admin] PATH_INFO: `/v3/users/300bbe096fcf4fabb645fbe2fbc075d4` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:13.522006 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bcaadb7-9c32-441e-9ddf-5015dce59330 None admin] RBAC: Authorizing `identity:get_user(user_id=300bbe096fcf4fabb645fbe2fbc075d4)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:13.523472 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bcaadb7-9c32-441e-9ddf-5015dce59330 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:13.525093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 542/1121] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:35:13 2022] GET /identity/v3/users/300bbe096fcf4fabb645fbe2fbc075d4 => generated 287 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:13.530929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99bc76d4-1601-48e2-a2f8-a1769fb0b1a1 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:13.638272 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99bc76d4-1601-48e2-a2f8-a1769fb0b1a1 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:13.662485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:13.674605 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d5a8ce8-5424-4d78-a6f8-f0b4cdfb2e13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:13.675246 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5a8ce8-5424-4d78-a6f8-f0b4cdfb2e13 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:13.675528 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5a8ce8-5424-4d78-a6f8-f0b4cdfb2e13 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:13.675834 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5a8ce8-5424-4d78-a6f8-f0b4cdfb2e13 None admin] PATH_INFO: `/v3/projects/3e9e9ad310a14001a620aa2149e177a5` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:13.676965 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d5a8ce8-5424-4d78-a6f8-f0b4cdfb2e13 None admin] RBAC: Authorizing `identity:get_project(project_id=3e9e9ad310a14001a620aa2149e177a5)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:13.678402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d5a8ce8-5424-4d78-a6f8-f0b4cdfb2e13 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:13.680093 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 580/1122] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:35:13 2022] GET /identity/v3/projects/3e9e9ad310a14001a620aa2149e177a5 => generated 312 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:13.686163 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bcaadb7-9c32-441e-9ddf-5015dce59330 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:13.786166 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bcaadb7-9c32-441e-9ddf-5015dce59330 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:13.820421 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7623d92-b7f7-48b8-bd46-24a1b44c68d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:13.820853 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7623d92-b7f7-48b8-bd46-24a1b44c68d5 None admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:13.821135 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7623d92-b7f7-48b8-bd46-24a1b44c68d5 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:13.821412 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7623d92-b7f7-48b8-bd46-24a1b44c68d5 None admin] PATH_INFO: `/v3/projects/3e9e9ad310a14001a620aa2149e177a5/users/300bbe096fcf4fabb645fbe2fbc075d4/roles/b8429ebd9c2a4cbbbbf30ad56ce23bfa` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:13.823648 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7623d92-b7f7-48b8-bd46-24a1b44c68d5 None admin] RBAC: Authorizing `identity:create_grant(project_id=3e9e9ad310a14001a620aa2149e177a5, user_id=300bbe096fcf4fabb645fbe2fbc075d4, role_id=b8429ebd9c2a4cbbbbf30ad56ce23bfa)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:13.825108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7623d92-b7f7-48b8-bd46-24a1b44c68d5 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:13.837874 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 543/1123] 10.222.0.102 () {58 vars in 1444 bytes} [Wed Aug 24 15:35:13 2022] PUT /identity/v3/projects/3e9e9ad310a14001a620aa2149e177a5/users/300bbe096fcf4fabb645fbe2fbc075d4/roles/b8429ebd9c2a4cbbbbf30ad56ce23bfa => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:35:15.740044 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9161e9d2-62d0-4445-8970-4afed8f0660d None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:15.740599 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9161e9d2-62d0-4445-8970-4afed8f0660d None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:15.740599 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9161e9d2-62d0-4445-8970-4afed8f0660d None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:15.741953 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 581/1124] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:35:15 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:35:15.747272 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cdae5e4-abc7-44ba-82bc-79f30a41b3dc None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:15.747712 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cdae5e4-abc7-44ba-82bc-79f30a41b3dc None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:15.747867 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cdae5e4-abc7-44ba-82bc-79f30a41b3dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:15.767499 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-5cdae5e4-abc7-44ba-82bc-79f30a41b3dc None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:35:15.768747 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-5cdae5e4-abc7-44ba-82bc-79f30a41b3dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:35:15.821874 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:15.840750 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 544/1125] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:35:15 2022] POST /identity/v3/auth/tokens => generated 2123 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:35:16.111122 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9161e9d2-62d0-4445-8970-4afed8f0660d None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:16.216491 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9161e9d2-62d0-4445-8970-4afed8f0660d None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:16.249708 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15dd0934-bc65-4d0b-9d98-65eb673eeeba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:16.250326 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dd0934-bc65-4d0b-9d98-65eb673eeeba None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:16.250602 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dd0934-bc65-4d0b-9d98-65eb673eeeba None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:16.250874 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dd0934-bc65-4d0b-9d98-65eb673eeeba None admin] PATH_INFO: `/v3/roles/b8429ebd9c2a4cbbbbf30ad56ce23bfa` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:16.253297 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15dd0934-bc65-4d0b-9d98-65eb673eeeba None admin] RBAC: Authorizing `identity:get_role(role_id=b8429ebd9c2a4cbbbbf30ad56ce23bfa)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:16.254659 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15dd0934-bc65-4d0b-9d98-65eb673eeeba None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:16.256178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 582/1126] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:35:16 2022] GET /identity/v3/roles/b8429ebd9c2a4cbbbbf30ad56ce23bfa => generated 224 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:16.261966 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cdae5e4-abc7-44ba-82bc-79f30a41b3dc None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:16.366393 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cdae5e4-abc7-44ba-82bc-79f30a41b3dc None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:16.400612 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d526d4c-7c3a-4bc8-a896-e41b6676b9a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:16.401239 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d526d4c-7c3a-4bc8-a896-e41b6676b9a6 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:16.401518 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d526d4c-7c3a-4bc8-a896-e41b6676b9a6 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:16.401792 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d526d4c-7c3a-4bc8-a896-e41b6676b9a6 None admin] PATH_INFO: `/v3/users/300bbe096fcf4fabb645fbe2fbc075d4` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:16.402930 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d526d4c-7c3a-4bc8-a896-e41b6676b9a6 None admin] RBAC: Authorizing `identity:get_user(user_id=300bbe096fcf4fabb645fbe2fbc075d4)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:16.404393 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d526d4c-7c3a-4bc8-a896-e41b6676b9a6 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:16.405969 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 545/1127] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:35:16 2022] GET /identity/v3/users/300bbe096fcf4fabb645fbe2fbc075d4 => generated 287 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:16.411806 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15dd0934-bc65-4d0b-9d98-65eb673eeeba None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:16.470642 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:16.510375 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15dd0934-bc65-4d0b-9d98-65eb673eeeba None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:16.543972 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2d9f66-22ea-48d2-81c0-d2ed871184e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:16.544610 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2d9f66-22ea-48d2-81c0-d2ed871184e8 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:16.544860 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2d9f66-22ea-48d2-81c0-d2ed871184e8 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:16.545132 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2d9f66-22ea-48d2-81c0-d2ed871184e8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:16.611428 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2d9f66-22ea-48d2-81c0-d2ed871184e8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:16.613038 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2d9f66-22ea-48d2-81c0-d2ed871184e8 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:16.669473 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:16.712167 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 583/1128] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:35:16 2022] GET /identity/v3/auth/tokens => generated 2123 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 15:35:16.718143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d526d4c-7c3a-4bc8-a896-e41b6676b9a6 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:16.772800 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:16.820211 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d526d4c-7c3a-4bc8-a896-e41b6676b9a6 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:16.854259 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34cb80db-5f0c-417c-9ba5-c758e76929d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:16.854883 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34cb80db-5f0c-417c-9ba5-c758e76929d4 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:16.855161 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34cb80db-5f0c-417c-9ba5-c758e76929d4 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:16.855431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34cb80db-5f0c-417c-9ba5-c758e76929d4 None admin] PATH_INFO: `/v3/projects/3e9e9ad310a14001a620aa2149e177a5` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:16.856585 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34cb80db-5f0c-417c-9ba5-c758e76929d4 None admin] RBAC: Authorizing `identity:get_project(project_id=3e9e9ad310a14001a620aa2149e177a5)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:16.858037 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34cb80db-5f0c-417c-9ba5-c758e76929d4 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:16.859745 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 546/1129] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:35:16 2022] GET /identity/v3/projects/3e9e9ad310a14001a620aa2149e177a5 => generated 312 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:35:16.866843 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2d9f66-22ea-48d2-81c0-d2ed871184e8 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:16.974539 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2d9f66-22ea-48d2-81c0-d2ed871184e8 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:17.009161 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221d7bc0-e903-463c-a598-99f0ab6aff8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:17.009765 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221d7bc0-e903-463c-a598-99f0ab6aff8a None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:17.010139 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221d7bc0-e903-463c-a598-99f0ab6aff8a None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:17.010316 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221d7bc0-e903-463c-a598-99f0ab6aff8a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:17.011046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-221d7bc0-e903-463c-a598-99f0ab6aff8a None admin] RBAC: Adding query filter params (role.id=b8429ebd9c2a4cbbbbf30ad56ce23bfa, scope.project.id=3e9e9ad310a14001a620aa2149e177a5, user.id=300bbe096fcf4fabb645fbe2fbc075d4) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:35:17.011355 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-221d7bc0-e903-463c-a598-99f0ab6aff8a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:17.012815 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-221d7bc0-e903-463c-a598-99f0ab6aff8a None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:17.019384 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 584/1130] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:35:16 2022] GET /identity/v3/role_assignments?user.id=300bbe096fcf4fabb645fbe2fbc075d4&scope.project.id=3e9e9ad310a14001a620aa2149e177a5&role.id=b8429ebd9c2a4cbbbbf30ad56ce23bfa => generated 610 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:18.966166 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1452cec-77bc-4319-ba31-6cc03d0101a2 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:18.966730 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1452cec-77bc-4319-ba31-6cc03d0101a2 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:18.966730 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1452cec-77bc-4319-ba31-6cc03d0101a2 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:18.968105 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 547/1131] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:35:18 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:35:18.974370 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79495287-bbd5-446b-9ba6-0b83b5894520 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:18.974831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79495287-bbd5-446b-9ba6-0b83b5894520 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:18.974920 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79495287-bbd5-446b-9ba6-0b83b5894520 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:18.994535 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-79495287-bbd5-446b-9ba6-0b83b5894520 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:35:18.995770 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-79495287-bbd5-446b-9ba6-0b83b5894520 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:35:19.014494 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:19.065687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 585/1132] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:35:18 2022] POST /identity/v3/auth/tokens => generated 2123 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:35:19.333859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1452cec-77bc-4319-ba31-6cc03d0101a2 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:19.444021 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1452cec-77bc-4319-ba31-6cc03d0101a2 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:19.464229 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:19.478503 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f3844e-da5a-45e8-a6dc-42f9d83db7b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:19.479132 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f3844e-da5a-45e8-a6dc-42f9d83db7b4 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:19.479414 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f3844e-da5a-45e8-a6dc-42f9d83db7b4 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:19.479712 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f3844e-da5a-45e8-a6dc-42f9d83db7b4 None admin] PATH_INFO: `/v3/domains/default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:19.480831 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f3844e-da5a-45e8-a6dc-42f9d83db7b4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:19.482269 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f3844e-da5a-45e8-a6dc-42f9d83db7b4 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:19.483872 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 548/1133] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:35:19 2022] GET /identity/v3/domains/default => generated 200 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:19.490922 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79495287-bbd5-446b-9ba6-0b83b5894520 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:19.587361 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79495287-bbd5-446b-9ba6-0b83b5894520 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:19.619985 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6878b1a1-0069-4e50-9a12-86fc81814674 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:19.620582 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6878b1a1-0069-4e50-9a12-86fc81814674 None admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:19.620862 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6878b1a1-0069-4e50-9a12-86fc81814674 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:19.621134 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6878b1a1-0069-4e50-9a12-86fc81814674 None admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:19.621791 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6878b1a1-0069-4e50-9a12-86fc81814674 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:19.623222 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6878b1a1-0069-4e50-9a12-86fc81814674 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:19.650505 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 586/1134] 10.222.0.102 () {60 vars in 1146 bytes} [Wed Aug 24 15:35:19 2022] POST /identity/v3/projects => generated 312 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:35:21.576043 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ac21a3-a6db-4f14-8674-64475a81bb8a None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:21.576576 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ac21a3-a6db-4f14-8674-64475a81bb8a None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:21.576576 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ac21a3-a6db-4f14-8674-64475a81bb8a None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:21.577981 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 549/1135] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:35:21 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:35:21.584312 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4cf4854-89f7-4f89-98bf-810c59315b8b None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:21.584741 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4cf4854-89f7-4f89-98bf-810c59315b8b None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:21.584869 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4cf4854-89f7-4f89-98bf-810c59315b8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:21.598566 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:21.604555 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-d4cf4854-89f7-4f89-98bf-810c59315b8b None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:35:21.606154 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-d4cf4854-89f7-4f89-98bf-810c59315b8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:35:21.671974 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 587/1136] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:35:21 2022] POST /identity/v3/auth/tokens => generated 2123 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:35:21.962183 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ac21a3-a6db-4f14-8674-64475a81bb8a None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:22.067724 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ac21a3-a6db-4f14-8674-64475a81bb8a None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:22.101551 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ac51cd-e93c-4740-aa73-4e5f4eb72da1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:22.102085 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ac51cd-e93c-4740-aa73-4e5f4eb72da1 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:22.102370 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ac51cd-e93c-4740-aa73-4e5f4eb72da1 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:22.102645 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ac51cd-e93c-4740-aa73-4e5f4eb72da1 None admin] PATH_INFO: `/v3/domains/default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:22.103777 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ac51cd-e93c-4740-aa73-4e5f4eb72da1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:22.105208 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ac51cd-e93c-4740-aa73-4e5f4eb72da1 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:22.106765 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 550/1137] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:35:21 2022] GET /identity/v3/domains/default => generated 200 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:22.113491 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4cf4854-89f7-4f89-98bf-810c59315b8b None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:22.221812 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4cf4854-89f7-4f89-98bf-810c59315b8b None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:22.257929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ace9244-63bc-4ee3-a5ba-10f488f2e192 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:22.257929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ace9244-63bc-4ee3-a5ba-10f488f2e192 None admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:22.258984 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ace9244-63bc-4ee3-a5ba-10f488f2e192 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:22.258984 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ace9244-63bc-4ee3-a5ba-10f488f2e192 None admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:22.259483 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ace9244-63bc-4ee3-a5ba-10f488f2e192 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:22.260956 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ace9244-63bc-4ee3-a5ba-10f488f2e192 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:22.271088 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:22.284955 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 588/1138] 10.222.0.102 () {60 vars in 1137 bytes} [Wed Aug 24 15:35:22 2022] POST /identity/v3/users => generated 287 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:35:24.279189 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fc84b6-4b83-4f57-b627-e757e7f7b9d5 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:24.279784 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fc84b6-4b83-4f57-b627-e757e7f7b9d5 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:24.279784 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fc84b6-4b83-4f57-b627-e757e7f7b9d5 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:24.281112 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 551/1139] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:35:24 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:35:24.288595 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e4d2c9-296c-4da6-af1c-791436408685 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:24.289112 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e4d2c9-296c-4da6-af1c-791436408685 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:24.289273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e4d2c9-296c-4da6-af1c-791436408685 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:24.309680 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-c4e4d2c9-296c-4da6-af1c-791436408685 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:35:24.310877 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-c4e4d2c9-296c-4da6-af1c-791436408685 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:35:24.378760 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 589/1140] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:35:24 2022] POST /identity/v3/auth/tokens => generated 2123 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 15:35:24.649950 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80fc84b6-4b83-4f57-b627-e757e7f7b9d5 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:24.707985 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:24.752471 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80fc84b6-4b83-4f57-b627-e757e7f7b9d5 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:24.786690 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1a6973-dc6e-4b6b-8ccc-1a338304f63e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:24.787308 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1a6973-dc6e-4b6b-8ccc-1a338304f63e None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:24.787988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1a6973-dc6e-4b6b-8ccc-1a338304f63e None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:24.787988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1a6973-dc6e-4b6b-8ccc-1a338304f63e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:24.797863 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd1a6973-dc6e-4b6b-8ccc-1a338304f63e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:35:24.802573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-dd1a6973-dc6e-4b6b-8ccc-1a338304f63e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:35:24.802573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:35:24.802573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:35:24.802573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:35:24.802573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:35:24.802573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:35:24.802573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:35:24.802573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:35:24.802573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:35:24.802573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:35:24.802573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:35:24.802573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:35:24.802573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:35:24.802573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:35:24.802573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:35:24.802573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:35:24.803695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:35:24.803695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:35:24.803695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:35:24.803695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:35:24.803695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:35:24.803695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:35:24.803695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:35:24.803695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:35:24.803695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:35:24.803695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:35:24.803695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:35:24.803695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:35:24.803695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:35:24.803695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:35:24.803695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:35:24.803695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:35:24.804629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:35:24.804629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:35:24.804629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:35:24.804629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:35:24.804629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:35:24.804629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:35:24.804629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:35:24.804629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:35:24.804629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:35:24.804629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:35:24.804629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:35:24.804629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:35:24.804629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 552/1141] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:35:24 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:35:24.810160 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e4d2c9-296c-4da6-af1c-791436408685 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:24.911769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e4d2c9-296c-4da6-af1c-791436408685 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:24.935215 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:24.949929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffb4a92-a25e-43c0-8010-502cf7eb2661 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:24.950548 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffb4a92-a25e-43c0-8010-502cf7eb2661 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:24.950832 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffb4a92-a25e-43c0-8010-502cf7eb2661 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:24.951111 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffb4a92-a25e-43c0-8010-502cf7eb2661 None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:24.951829 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffb4a92-a25e-43c0-8010-502cf7eb2661 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:35:24.952164 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffb4a92-a25e-43c0-8010-502cf7eb2661 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:24.954352 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffb4a92-a25e-43c0-8010-502cf7eb2661 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:24.964169 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 590/1142] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:35:24 2022] GET /identity/v3/roles?name=admin => generated 341 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:24.970315 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1a6973-dc6e-4b6b-8ccc-1a338304f63e None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:25.078224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1a6973-dc6e-4b6b-8ccc-1a338304f63e None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:25.095304 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:25.114268 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44335be2-b986-4fc7-8563-ca4bebfe3691 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:25.114889 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44335be2-b986-4fc7-8563-ca4bebfe3691 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:25.115172 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44335be2-b986-4fc7-8563-ca4bebfe3691 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:25.115450 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44335be2-b986-4fc7-8563-ca4bebfe3691 None admin] PATH_INFO: `/v3/users/f0770018ab2742ac847382607ab31d8c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:25.129130 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44335be2-b986-4fc7-8563-ca4bebfe3691 None admin] RBAC: Authorizing `identity:get_user(user_id=f0770018ab2742ac847382607ab31d8c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:25.130603 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44335be2-b986-4fc7-8563-ca4bebfe3691 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:25.132252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 553/1143] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:35:24 2022] GET /identity/v3/users/f0770018ab2742ac847382607ab31d8c => generated 287 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:25.138204 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffb4a92-a25e-43c0-8010-502cf7eb2661 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:25.237964 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffb4a92-a25e-43c0-8010-502cf7eb2661 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:25.271275 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbcfb0e-19fd-467a-bda9-d2be25503bed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:25.271916 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbcfb0e-19fd-467a-bda9-d2be25503bed None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:25.272195 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbcfb0e-19fd-467a-bda9-d2be25503bed None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:25.272467 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbcfb0e-19fd-467a-bda9-d2be25503bed None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:25.324603 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:25.340458 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cbcfb0e-19fd-467a-bda9-d2be25503bed None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:25.341908 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cbcfb0e-19fd-467a-bda9-d2be25503bed None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:25.438404 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 591/1144] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:35:25 2022] GET /identity/v3/auth/tokens => generated 2123 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 15:35:25.445353 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44335be2-b986-4fc7-8563-ca4bebfe3691 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:25.545211 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44335be2-b986-4fc7-8563-ca4bebfe3691 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:25.579312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f32949-e5c3-40c4-95eb-fc39e5112659 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:25.579953 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f32949-e5c3-40c4-95eb-fc39e5112659 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:25.580229 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f32949-e5c3-40c4-95eb-fc39e5112659 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:25.580496 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f32949-e5c3-40c4-95eb-fc39e5112659 None admin] PATH_INFO: `/v3/projects/cf34222322b3470cabc8966b22bf576b` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:25.581602 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f32949-e5c3-40c4-95eb-fc39e5112659 None admin] RBAC: Authorizing `identity:get_project(project_id=cf34222322b3470cabc8966b22bf576b)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:25.583019 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f32949-e5c3-40c4-95eb-fc39e5112659 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:25.584664 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 554/1145] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:35:25 2022] GET /identity/v3/projects/cf34222322b3470cabc8966b22bf576b => generated 312 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:25.590775 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbcfb0e-19fd-467a-bda9-d2be25503bed None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:25.673618 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:25.691085 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbcfb0e-19fd-467a-bda9-d2be25503bed None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:25.725027 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3506c10b-92fb-4de6-957a-b2fb3c80678c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:25.725636 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3506c10b-92fb-4de6-957a-b2fb3c80678c None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:25.725918 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3506c10b-92fb-4de6-957a-b2fb3c80678c None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:25.726193 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3506c10b-92fb-4de6-957a-b2fb3c80678c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:25.726921 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3506c10b-92fb-4de6-957a-b2fb3c80678c None admin] RBAC: Adding query filter params (role.id=4d8fbbaea81a4145b89af12350bb067c, scope.project.id=cf34222322b3470cabc8966b22bf576b, user.id=f0770018ab2742ac847382607ab31d8c) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:35:25.727236 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3506c10b-92fb-4de6-957a-b2fb3c80678c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:25.728692 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3506c10b-92fb-4de6-957a-b2fb3c80678c None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:25.734076 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 592/1146] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:35:25 2022] GET /identity/v3/role_assignments?user.id=f0770018ab2742ac847382607ab31d8c&scope.project.id=cf34222322b3470cabc8966b22bf576b&role.id=4d8fbbaea81a4145b89af12350bb067c => generated 260 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:27.652072 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6e9c08-bbe3-43a9-9695-09cad6c3a5f0 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:27.652619 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6e9c08-bbe3-43a9-9695-09cad6c3a5f0 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:27.652619 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6e9c08-bbe3-43a9-9695-09cad6c3a5f0 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:27.653984 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 555/1147] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:35:27 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:35:27.660190 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45abbdde-dedc-43f4-b417-75cb52dd2bf0 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:27.660627 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45abbdde-dedc-43f4-b417-75cb52dd2bf0 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:27.660761 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45abbdde-dedc-43f4-b417-75cb52dd2bf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:27.678508 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-45abbdde-dedc-43f4-b417-75cb52dd2bf0 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:35:27.679735 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-45abbdde-dedc-43f4-b417-75cb52dd2bf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:35:27.748166 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 593/1148] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:35:27 2022] POST /identity/v3/auth/tokens => generated 2123 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:35:28.015947 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6e9c08-bbe3-43a9-9695-09cad6c3a5f0 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:28.061027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:28.119323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6e9c08-bbe3-43a9-9695-09cad6c3a5f0 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:28.154964 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e435ab35-e85b-4c96-8763-5dc518a6557a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:28.155744 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e435ab35-e85b-4c96-8763-5dc518a6557a None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:28.156247 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e435ab35-e85b-4c96-8763-5dc518a6557a None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:28.156741 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e435ab35-e85b-4c96-8763-5dc518a6557a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:28.169789 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e435ab35-e85b-4c96-8763-5dc518a6557a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:35:28.175236 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-e435ab35-e85b-4c96-8763-5dc518a6557a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:35:28.175236 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:35:28.175236 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:35:28.175236 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:35:28.175236 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:35:28.175236 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:35:28.175236 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:35:28.175236 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:35:28.175236 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:35:28.175236 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:35:28.175236 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:35:28.175236 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:35:28.175236 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:35:28.175236 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:35:28.175236 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:35:28.175236 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:35:28.176411 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:35:28.176411 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:35:28.176411 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:35:28.176411 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:35:28.176411 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:35:28.176411 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:35:28.176411 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:35:28.176411 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:35:28.176411 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:35:28.176411 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:35:28.176411 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:35:28.176411 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:35:28.176411 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:35:28.176411 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:35:28.176411 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:35:28.176411 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:35:28.177476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:35:28.177476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:35:28.177476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:35:28.177476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:35:28.177476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:35:28.177476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:35:28.177476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:35:28.177476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:35:28.177476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:35:28.177476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:35:28.177476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:35:28.177476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:35:28.177476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 556/1149] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:35:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:35:28.182785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45abbdde-dedc-43f4-b417-75cb52dd2bf0 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:28.262058 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:28.284330 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45abbdde-dedc-43f4-b417-75cb52dd2bf0 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:28.318721 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-140f78d5-18db-46e8-9cb8-0fb1d0399b6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:28.319327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140f78d5-18db-46e8-9cb8-0fb1d0399b6e None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:28.319605 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140f78d5-18db-46e8-9cb8-0fb1d0399b6e None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:28.319906 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140f78d5-18db-46e8-9cb8-0fb1d0399b6e None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:28.320838 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-140f78d5-18db-46e8-9cb8-0fb1d0399b6e None admin] RBAC: Adding query filter params (name=admin) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:35:28.321140 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-140f78d5-18db-46e8-9cb8-0fb1d0399b6e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:28.322513 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-140f78d5-18db-46e8-9cb8-0fb1d0399b6e None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:28.331176 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 594/1150] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:35:28 2022] GET /identity/v3/roles?name=admin => generated 341 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:28.338435 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e435ab35-e85b-4c96-8763-5dc518a6557a None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:28.442029 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e435ab35-e85b-4c96-8763-5dc518a6557a None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:28.443566 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:28.477707 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d978cbc-4c2a-4fcf-84c3-be9b0447f9a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:28.478326 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d978cbc-4c2a-4fcf-84c3-be9b0447f9a6 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:28.478604 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d978cbc-4c2a-4fcf-84c3-be9b0447f9a6 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:28.478880 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d978cbc-4c2a-4fcf-84c3-be9b0447f9a6 None admin] PATH_INFO: `/v3/users/f0770018ab2742ac847382607ab31d8c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:28.480034 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d978cbc-4c2a-4fcf-84c3-be9b0447f9a6 None admin] RBAC: Authorizing `identity:get_user(user_id=f0770018ab2742ac847382607ab31d8c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:28.481506 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d978cbc-4c2a-4fcf-84c3-be9b0447f9a6 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:28.483086 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 557/1151] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:35:28 2022] GET /identity/v3/users/f0770018ab2742ac847382607ab31d8c => generated 287 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:28.489703 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-140f78d5-18db-46e8-9cb8-0fb1d0399b6e None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:28.589814 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-140f78d5-18db-46e8-9cb8-0fb1d0399b6e None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:28.623868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2df925-ffe5-4200-b542-8c30c80c18df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:28.624491 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2df925-ffe5-4200-b542-8c30c80c18df None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:28.624780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2df925-ffe5-4200-b542-8c30c80c18df None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:28.625059 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2df925-ffe5-4200-b542-8c30c80c18df None admin] PATH_INFO: `/v3/projects/cf34222322b3470cabc8966b22bf576b` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:28.626178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab2df925-ffe5-4200-b542-8c30c80c18df None admin] RBAC: Authorizing `identity:get_project(project_id=cf34222322b3470cabc8966b22bf576b)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:28.627862 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab2df925-ffe5-4200-b542-8c30c80c18df None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:28.629525 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 595/1152] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:35:28 2022] GET /identity/v3/projects/cf34222322b3470cabc8966b22bf576b => generated 312 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:28.635598 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d978cbc-4c2a-4fcf-84c3-be9b0447f9a6 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:28.739686 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d978cbc-4c2a-4fcf-84c3-be9b0447f9a6 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:28.775195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1201f4b5-20b1-4fe7-a0c1-a68e15dd9e77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:28.775877 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1201f4b5-20b1-4fe7-a0c1-a68e15dd9e77 None admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:28.776160 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1201f4b5-20b1-4fe7-a0c1-a68e15dd9e77 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:28.776438 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1201f4b5-20b1-4fe7-a0c1-a68e15dd9e77 None admin] PATH_INFO: `/v3/projects/cf34222322b3470cabc8966b22bf576b/users/f0770018ab2742ac847382607ab31d8c/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:28.778384 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1201f4b5-20b1-4fe7-a0c1-a68e15dd9e77 None admin] RBAC: Authorizing `identity:create_grant(project_id=cf34222322b3470cabc8966b22bf576b, user_id=f0770018ab2742ac847382607ab31d8c, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:28.779899 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1201f4b5-20b1-4fe7-a0c1-a68e15dd9e77 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:28.794830 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 558/1153] 10.222.0.102 () {58 vars in 1444 bytes} [Wed Aug 24 15:35:28 2022] PUT /identity/v3/projects/cf34222322b3470cabc8966b22bf576b/users/f0770018ab2742ac847382607ab31d8c/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:35:30.738990 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a234ad1-b471-4017-b31e-d30880f3878f None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:30.739558 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a234ad1-b471-4017-b31e-d30880f3878f None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:30.739558 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a234ad1-b471-4017-b31e-d30880f3878f None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:30.740915 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 596/1154] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:35:30 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:35:30.746236 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4c60ae-24e9-42b6-9375-f539d295ce83 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:30.746662 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4c60ae-24e9-42b6-9375-f539d295ce83 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:30.746786 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4c60ae-24e9-42b6-9375-f539d295ce83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:30.765716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-af4c60ae-24e9-42b6-9375-f539d295ce83 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:35:30.766897 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-af4c60ae-24e9-42b6-9375-f539d295ce83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:35:30.790800 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:30.837921 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 559/1155] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:35:30 2022] POST /identity/v3/auth/tokens => generated 2123 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:35:31.115102 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a234ad1-b471-4017-b31e-d30880f3878f None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:31.126394 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:31.222700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a234ad1-b471-4017-b31e-d30880f3878f None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:31.261904 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9998f48e-4a83-4fa1-9574-8ab6b2d5a711 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:31.262539 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9998f48e-4a83-4fa1-9574-8ab6b2d5a711 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:31.262824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9998f48e-4a83-4fa1-9574-8ab6b2d5a711 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:31.263253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9998f48e-4a83-4fa1-9574-8ab6b2d5a711 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:31.278374 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9998f48e-4a83-4fa1-9574-8ab6b2d5a711 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:35:31.283258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-9998f48e-4a83-4fa1-9574-8ab6b2d5a711 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:35:31.283258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:35:31.283258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:35:31.283258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:35:31.283258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:35:31.283258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:35:31.283258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:35:31.283258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:35:31.283258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:35:31.283258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:35:31.283258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:35:31.283258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:35:31.283258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:35:31.283258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:35:31.283258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:35:31.283258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:35:31.284967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:35:31.284967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:35:31.284967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:35:31.284967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:35:31.284967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:35:31.284967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:35:31.284967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:35:31.284967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:35:31.284967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:35:31.284967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:35:31.284967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:35:31.284967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:35:31.284967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:35:31.284967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:35:31.284967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:35:31.284967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:35:31.285886 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:35:31.285886 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:35:31.285886 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:35:31.285886 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:35:31.285886 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:35:31.285886 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:35:31.285886 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:35:31.285886 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:35:31.285886 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:35:31.285886 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:35:31.285886 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:35:31.285886 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:35:31.285886 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 597/1156] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:35:31 2022] GET /identity/v3/roles/admin => generated 83 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:35:31.291537 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4c60ae-24e9-42b6-9375-f539d295ce83 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:31.400361 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4c60ae-24e9-42b6-9375-f539d295ce83 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:31.433988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5bdbfd-783e-4dcd-98cf-2fa7f3b027ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:31.434578 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5bdbfd-783e-4dcd-98cf-2fa7f3b027ad None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:31.434856 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5bdbfd-783e-4dcd-98cf-2fa7f3b027ad None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:31.435127 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5bdbfd-783e-4dcd-98cf-2fa7f3b027ad None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:31.436105 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5bdbfd-783e-4dcd-98cf-2fa7f3b027ad None admin] RBAC: Adding query filter params (name=admin) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:35:31.436105 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5bdbfd-783e-4dcd-98cf-2fa7f3b027ad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:31.437449 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5bdbfd-783e-4dcd-98cf-2fa7f3b027ad None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:31.445732 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 560/1157] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:35:31 2022] GET /identity/v3/roles?name=admin => generated 341 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:31.451557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9998f48e-4a83-4fa1-9574-8ab6b2d5a711 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:31.519445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:31.552967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9998f48e-4a83-4fa1-9574-8ab6b2d5a711 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:31.588892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a5fdb3-5d0f-4f8c-9584-c1e9a6d0afdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:31.589550 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a5fdb3-5d0f-4f8c-9584-c1e9a6d0afdb None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:31.589834 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a5fdb3-5d0f-4f8c-9584-c1e9a6d0afdb None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:31.590111 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a5fdb3-5d0f-4f8c-9584-c1e9a6d0afdb None admin] PATH_INFO: `/v3/users/f0770018ab2742ac847382607ab31d8c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:31.591292 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a5fdb3-5d0f-4f8c-9584-c1e9a6d0afdb None admin] RBAC: Authorizing `identity:get_user(user_id=f0770018ab2742ac847382607ab31d8c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:31.592803 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a5fdb3-5d0f-4f8c-9584-c1e9a6d0afdb None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:31.594640 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 598/1158] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:35:31 2022] GET /identity/v3/users/f0770018ab2742ac847382607ab31d8c => generated 287 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:31.605431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5bdbfd-783e-4dcd-98cf-2fa7f3b027ad None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:31.633549 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:31.714133 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5bdbfd-783e-4dcd-98cf-2fa7f3b027ad None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:31.748094 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f2ee2d-880a-4828-bbca-76a08ca7ee95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:31.748735 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f2ee2d-880a-4828-bbca-76a08ca7ee95 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:31.748953 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f2ee2d-880a-4828-bbca-76a08ca7ee95 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:31.749287 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f2ee2d-880a-4828-bbca-76a08ca7ee95 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:31.816949 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f2ee2d-880a-4828-bbca-76a08ca7ee95 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:31.818391 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f2ee2d-880a-4828-bbca-76a08ca7ee95 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:31.843553 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:31.920363 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 561/1159] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:35:31 2022] GET /identity/v3/auth/tokens => generated 2123 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 15:35:31.927726 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a5fdb3-5d0f-4f8c-9584-c1e9a6d0afdb None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:32.032264 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a5fdb3-5d0f-4f8c-9584-c1e9a6d0afdb None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:32.067542 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc6d1af-7482-4524-aa7f-7edc82efa55c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:32.068204 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc6d1af-7482-4524-aa7f-7edc82efa55c None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:32.068487 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc6d1af-7482-4524-aa7f-7edc82efa55c None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:32.068762 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc6d1af-7482-4524-aa7f-7edc82efa55c None admin] PATH_INFO: `/v3/projects/cf34222322b3470cabc8966b22bf576b` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:32.069902 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc6d1af-7482-4524-aa7f-7edc82efa55c None admin] RBAC: Authorizing `identity:get_project(project_id=cf34222322b3470cabc8966b22bf576b)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:32.071613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc6d1af-7482-4524-aa7f-7edc82efa55c None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:32.073287 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 599/1160] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:35:31 2022] GET /identity/v3/projects/cf34222322b3470cabc8966b22bf576b => generated 312 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:35:32.079415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f2ee2d-880a-4828-bbca-76a08ca7ee95 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:32.179453 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f2ee2d-880a-4828-bbca-76a08ca7ee95 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:32.202468 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:32.214909 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5af9856-b789-447e-bfc7-ac75708c4c36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:32.215506 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5af9856-b789-447e-bfc7-ac75708c4c36 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:32.215835 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5af9856-b789-447e-bfc7-ac75708c4c36 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:32.216089 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5af9856-b789-447e-bfc7-ac75708c4c36 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:32.216814 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5af9856-b789-447e-bfc7-ac75708c4c36 None admin] RBAC: Adding query filter params (role.id=4d8fbbaea81a4145b89af12350bb067c, scope.project.id=cf34222322b3470cabc8966b22bf576b, user.id=f0770018ab2742ac847382607ab31d8c) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:35:32.217116 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5af9856-b789-447e-bfc7-ac75708c4c36 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:32.218530 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5af9856-b789-447e-bfc7-ac75708c4c36 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:32.224698 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 562/1161] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:35:32 2022] GET /identity/v3/role_assignments?user.id=f0770018ab2742ac847382607ab31d8c&scope.project.id=cf34222322b3470cabc8966b22bf576b&role.id=4d8fbbaea81a4145b89af12350bb067c => generated 610 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:34.185014 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7d0361-5c11-481c-b2d1-489905f5511c None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:34.185565 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7d0361-5c11-481c-b2d1-489905f5511c None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:34.185565 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7d0361-5c11-481c-b2d1-489905f5511c None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:34.186905 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 600/1162] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:35:34 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:35:34.192453 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af247985-334c-4795-8d8a-7e28f75d5908 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:34.192921 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af247985-334c-4795-8d8a-7e28f75d5908 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:34.193124 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af247985-334c-4795-8d8a-7e28f75d5908 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:34.212096 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-af247985-334c-4795-8d8a-7e28f75d5908 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:35:34.213291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-af247985-334c-4795-8d8a-7e28f75d5908 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:35:34.281363 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 563/1163] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:35:34 2022] POST /identity/v3/auth/tokens => generated 2123 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:35:34.556898 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7d0361-5c11-481c-b2d1-489905f5511c None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:34.560734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:34.657827 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7d0361-5c11-481c-b2d1-489905f5511c None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:34.690913 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c610350-f782-4664-a278-1f6dfa5a9217 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:34.691509 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c610350-f782-4664-a278-1f6dfa5a9217 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:34.691818 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c610350-f782-4664-a278-1f6dfa5a9217 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:34.692092 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c610350-f782-4664-a278-1f6dfa5a9217 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:34.759164 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c610350-f782-4664-a278-1f6dfa5a9217 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:34.760631 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c610350-f782-4664-a278-1f6dfa5a9217 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:34.763207 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:34.862695 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 601/1164] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:35:34 2022] GET /identity/v3/auth/tokens => generated 2123 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 15:35:34.868485 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af247985-334c-4795-8d8a-7e28f75d5908 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:34.971770 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af247985-334c-4795-8d8a-7e28f75d5908 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:34.988866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:35.007882 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1654ae98-2746-48e2-861e-5566ee9fdd3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:35.008533 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1654ae98-2746-48e2-861e-5566ee9fdd3d None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:35.008816 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1654ae98-2746-48e2-861e-5566ee9fdd3d None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:35.009090 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1654ae98-2746-48e2-861e-5566ee9fdd3d None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:35.015805 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1654ae98-2746-48e2-861e-5566ee9fdd3d None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:35.017257 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1654ae98-2746-48e2-861e-5566ee9fdd3d None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:35.029292 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-1654ae98-2746-48e2-861e-5566ee9fdd3d None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Aug 24 15:35:35.029292 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:35:35.029292 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Aug 24 15:35:35.029292 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Aug 24 15:35:35.029292 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:35:35.029292 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:35:35.029292 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:35:35.029292 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:35:35.029292 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Aug 24 15:35:35.029292 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:35:35.029292 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 15:35:35.029292 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:35:35.029292 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:35:35.029292 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:35:35.029292 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:35:35.029292 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:35:35.029292 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:35:35.030567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:35:35.030567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:35:35.030567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:35:35.030567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:35:35.030567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:35:35.030567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:35:35.030567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:35:35.030567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:35:35.030567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:35:35.030567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:35:35.030567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:35:35.030567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:35:35.030567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:35:35.030567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:35:35.030567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:35:35.030567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:35:35.031507 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:35:35.031507 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:35:35.031507 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:35:35.031507 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:35:35.031507 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:35:35.031507 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:35:35.031507 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:35:35.031507 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:35:35.031507 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:35:35.031507 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:35:35.031507 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Aug 24 15:35:35.031507 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:35:35.031507 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Aug 24 15:35:35.031507 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:35:35.031507 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 564/1165] 10.222.0.102 () {56 vars in 1122 bytes} [Wed Aug 24 15:35:34 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:35:35.037495 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c610350-f782-4664-a278-1f6dfa5a9217 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:35.139076 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c610350-f782-4664-a278-1f6dfa5a9217 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:35.141383 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:35.176463 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-285f7e76-5bd7-475e-ac50-d2f3e78fc2ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:35.177061 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285f7e76-5bd7-475e-ac50-d2f3e78fc2ce None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:35.177342 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285f7e76-5bd7-475e-ac50-d2f3e78fc2ce None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:35.177617 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285f7e76-5bd7-475e-ac50-d2f3e78fc2ce None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:35.178265 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-285f7e76-5bd7-475e-ac50-d2f3e78fc2ce None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:35:35.178574 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-285f7e76-5bd7-475e-ac50-d2f3e78fc2ce None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:35.179992 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-285f7e76-5bd7-475e-ac50-d2f3e78fc2ce None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:35.187182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 602/1166] 10.222.0.102 () {56 vars in 1120 bytes} [Wed Aug 24 15:35:35 2022] GET /identity/v3/domains?name=swift_test => generated 126 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:37.092877 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e37d6b-543e-4d48-87ff-bf4da5d88ab6 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:37.093442 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e37d6b-543e-4d48-87ff-bf4da5d88ab6 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:37.093442 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e37d6b-543e-4d48-87ff-bf4da5d88ab6 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:37.094736 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 565/1167] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:35:37 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:35:37.100064 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2d86f9-bc2d-4b3c-a47c-ce485add1347 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:37.100452 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2d86f9-bc2d-4b3c-a47c-ce485add1347 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:37.100550 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2d86f9-bc2d-4b3c-a47c-ce485add1347 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:37.118881 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-ba2d86f9-bc2d-4b3c-a47c-ce485add1347 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:35:37.120083 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-ba2d86f9-bc2d-4b3c-a47c-ce485add1347 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:35:37.186144 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 603/1168] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:35:37 2022] POST /identity/v3/auth/tokens => generated 2123 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 15:35:37.468071 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e37d6b-543e-4d48-87ff-bf4da5d88ab6 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:37.575426 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e37d6b-543e-4d48-87ff-bf4da5d88ab6 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:37.614552 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600b69c7-5d43-4102-a961-be9e0f4bc3ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:37.615153 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600b69c7-5d43-4102-a961-be9e0f4bc3ea None admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:37.615438 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600b69c7-5d43-4102-a961-be9e0f4bc3ea None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:37.615849 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600b69c7-5d43-4102-a961-be9e0f4bc3ea None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:37.616522 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600b69c7-5d43-4102-a961-be9e0f4bc3ea None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:37.617875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600b69c7-5d43-4102-a961-be9e0f4bc3ea None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:37.633561 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 566/1169] 10.222.0.102 () {60 vars in 1143 bytes} [Wed Aug 24 15:35:37 2022] POST /identity/v3/domains => generated 268 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:35:39.559388 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0233fdbd-ed84-4258-a6a0-488701c3bc1b None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:39.559993 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0233fdbd-ed84-4258-a6a0-488701c3bc1b None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:39.559993 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0233fdbd-ed84-4258-a6a0-488701c3bc1b None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:39.561298 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 604/1170] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:35:39 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:35:39.567276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbea13fd-6866-48b9-b894-5292b9a556d5 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:39.567713 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbea13fd-6866-48b9-b894-5292b9a556d5 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:39.567859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbea13fd-6866-48b9-b894-5292b9a556d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:39.586559 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-fbea13fd-6866-48b9-b894-5292b9a556d5 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:35:39.587771 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-fbea13fd-6866-48b9-b894-5292b9a556d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:35:39.618239 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:39.664066 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 567/1171] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:35:39 2022] POST /identity/v3/auth/tokens => generated 2123 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:35:39.930305 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0233fdbd-ed84-4258-a6a0-488701c3bc1b None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:40.037231 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:40.042665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0233fdbd-ed84-4258-a6a0-488701c3bc1b None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:40.078501 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a59f11-411c-479b-a989-2c7f9b8effa7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:40.079310 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a59f11-411c-479b-a989-2c7f9b8effa7 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:40.079595 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a59f11-411c-479b-a989-2c7f9b8effa7 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:40.079902 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a59f11-411c-479b-a989-2c7f9b8effa7 None admin] PATH_INFO: `/v3/domains/427cf352b83b4ca0b31cf44c5ea26288` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:40.090142 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a59f11-411c-479b-a989-2c7f9b8effa7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=427cf352b83b4ca0b31cf44c5ea26288)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:40.091594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a59f11-411c-479b-a989-2c7f9b8effa7 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:40.093232 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 605/1172] 10.222.0.102 () {56 vars in 1188 bytes} [Wed Aug 24 15:35:39 2022] GET /identity/v3/domains/427cf352b83b4ca0b31cf44c5ea26288 => generated 268 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:40.100146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbea13fd-6866-48b9-b894-5292b9a556d5 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:40.203777 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbea13fd-6866-48b9-b894-5292b9a556d5 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:40.239719 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5da379-e62b-4898-9fd0-3fee260726a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:40.240315 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5da379-e62b-4898-9fd0-3fee260726a6 None admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:40.240601 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5da379-e62b-4898-9fd0-3fee260726a6 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:40.240874 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5da379-e62b-4898-9fd0-3fee260726a6 None admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:40.241519 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5da379-e62b-4898-9fd0-3fee260726a6 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:40.242949 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5da379-e62b-4898-9fd0-3fee260726a6 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:40.270305 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 568/1173] 10.222.0.102 () {60 vars in 1146 bytes} [Wed Aug 24 15:35:40 2022] POST /identity/v3/projects => generated 362 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:35:42.236161 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97e2e2c-582e-4263-a1bc-76301da3fe3e None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:42.236707 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97e2e2c-582e-4263-a1bc-76301da3fe3e None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:42.236707 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97e2e2c-582e-4263-a1bc-76301da3fe3e None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:42.238077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 606/1174] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:35:42 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:35:42.243409 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bdd517-32c8-4b08-b446-a518ab09f108 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:42.243877 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bdd517-32c8-4b08-b446-a518ab09f108 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:42.243977 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bdd517-32c8-4b08-b446-a518ab09f108 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:42.262531 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-76bdd517-32c8-4b08-b446-a518ab09f108 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:35:42.263747 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-76bdd517-32c8-4b08-b446-a518ab09f108 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:35:42.277772 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:42.333698 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 569/1175] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:35:42 2022] POST /identity/v3/auth/tokens => generated 2123 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:35:42.606472 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97e2e2c-582e-4263-a1bc-76301da3fe3e None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:42.714382 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97e2e2c-582e-4263-a1bc-76301da3fe3e None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:42.748593 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7231a256-6385-48c1-b8cf-f4c6c33a6855 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:42.749220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7231a256-6385-48c1-b8cf-f4c6c33a6855 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:42.749505 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7231a256-6385-48c1-b8cf-f4c6c33a6855 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:42.749782 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7231a256-6385-48c1-b8cf-f4c6c33a6855 None admin] PATH_INFO: `/v3/domains/427cf352b83b4ca0b31cf44c5ea26288` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:42.750981 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7231a256-6385-48c1-b8cf-f4c6c33a6855 None admin] RBAC: Authorizing `identity:get_domain(domain_id=427cf352b83b4ca0b31cf44c5ea26288)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:42.752442 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7231a256-6385-48c1-b8cf-f4c6c33a6855 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:42.754038 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 607/1176] 10.222.0.102 () {56 vars in 1188 bytes} [Wed Aug 24 15:35:42 2022] GET /identity/v3/domains/427cf352b83b4ca0b31cf44c5ea26288 => generated 268 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:42.759866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bdd517-32c8-4b08-b446-a518ab09f108 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:42.863847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bdd517-32c8-4b08-b446-a518ab09f108 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:42.904777 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b5ead9-73af-458e-bcf6-0b4ebf9badf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:42.905378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b5ead9-73af-458e-bcf6-0b4ebf9badf1 None admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:42.905661 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b5ead9-73af-458e-bcf6-0b4ebf9badf1 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:42.905934 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b5ead9-73af-458e-bcf6-0b4ebf9badf1 None admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:42.907508 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b5ead9-73af-458e-bcf6-0b4ebf9badf1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:42.908970 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b5ead9-73af-458e-bcf6-0b4ebf9badf1 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:42.932382 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 570/1177] 10.222.0.102 () {60 vars in 1137 bytes} [Wed Aug 24 15:35:42 2022] POST /identity/v3/users => generated 312 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:35:44.897163 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe6e496-5572-4505-a4f4-ebd44036c1e9 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:44.897672 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe6e496-5572-4505-a4f4-ebd44036c1e9 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:44.897787 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe6e496-5572-4505-a4f4-ebd44036c1e9 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:44.899063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 608/1178] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:35:44 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:35:44.904268 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19843f00-ba7f-4620-8b9e-ed4699558d20 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:44.904674 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19843f00-ba7f-4620-8b9e-ed4699558d20 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:44.904832 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19843f00-ba7f-4620-8b9e-ed4699558d20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:44.912285 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:44.925468 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-19843f00-ba7f-4620-8b9e-ed4699558d20 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:35:44.926671 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-19843f00-ba7f-4620-8b9e-ed4699558d20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:35:44.997284 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 571/1179] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:35:44 2022] POST /identity/v3/auth/tokens => generated 2123 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:35:45.266062 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe6e496-5572-4505-a4f4-ebd44036c1e9 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:45.294890 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:45.369745 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe6e496-5572-4505-a4f4-ebd44036c1e9 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:45.404048 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a029b86d-3df8-4f52-a10c-01f41f4b48f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:45.404670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a029b86d-3df8-4f52-a10c-01f41f4b48f3 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:45.404946 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a029b86d-3df8-4f52-a10c-01f41f4b48f3 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:45.405212 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a029b86d-3df8-4f52-a10c-01f41f4b48f3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:45.416585 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a029b86d-3df8-4f52-a10c-01f41f4b48f3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:35:45.421294 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-a029b86d-3df8-4f52-a10c-01f41f4b48f3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:35:45.421294 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:35:45.421294 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:35:45.421294 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:35:45.421294 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:35:45.421294 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:35:45.421294 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:35:45.421294 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:35:45.421294 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:35:45.421294 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:35:45.421294 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:35:45.421294 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:35:45.421294 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:35:45.421294 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:35:45.421294 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:35:45.421294 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:35:45.422403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:35:45.422403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:35:45.422403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:35:45.422403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:35:45.422403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:35:45.422403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:35:45.422403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:35:45.422403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:35:45.422403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:35:45.422403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:35:45.422403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:35:45.422403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:35:45.422403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:35:45.422403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:35:45.422403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:35:45.422403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:35:45.423460 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:35:45.423460 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:35:45.423460 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:35:45.423460 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:35:45.423460 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:35:45.423460 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:35:45.423460 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:35:45.423460 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:35:45.423460 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:35:45.423460 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:35:45.423460 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:35:45.423460 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:35:45.423460 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 609/1180] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:35:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:35:45.428471 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19843f00-ba7f-4620-8b9e-ed4699558d20 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:45.526298 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19843f00-ba7f-4620-8b9e-ed4699558d20 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:45.560167 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acce440e-0ad5-440a-ad5c-bdb66a91aaad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:45.560773 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acce440e-0ad5-440a-ad5c-bdb66a91aaad None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:45.561042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acce440e-0ad5-440a-ad5c-bdb66a91aaad None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:45.561312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acce440e-0ad5-440a-ad5c-bdb66a91aaad None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:45.562002 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acce440e-0ad5-440a-ad5c-bdb66a91aaad None admin] RBAC: Adding query filter params (name=admin) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:35:45.562299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acce440e-0ad5-440a-ad5c-bdb66a91aaad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:45.563833 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acce440e-0ad5-440a-ad5c-bdb66a91aaad None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:45.565196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:45.573217 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 572/1181] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:35:45 2022] GET /identity/v3/roles?name=admin => generated 341 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:45.580007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a029b86d-3df8-4f52-a10c-01f41f4b48f3 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:45.668041 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:45.683362 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a029b86d-3df8-4f52-a10c-01f41f4b48f3 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:45.717084 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39670681-00af-437b-9aa4-ac800beef98f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:45.717707 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39670681-00af-437b-9aa4-ac800beef98f None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:45.718674 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39670681-00af-437b-9aa4-ac800beef98f None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:45.718991 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39670681-00af-437b-9aa4-ac800beef98f None admin] PATH_INFO: `/v3/users/596965ec955a4b989dc27603813d633c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:45.731951 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39670681-00af-437b-9aa4-ac800beef98f None admin] RBAC: Authorizing `identity:get_user(user_id=596965ec955a4b989dc27603813d633c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:45.733399 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39670681-00af-437b-9aa4-ac800beef98f None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:45.735185 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 610/1182] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:35:45 2022] GET /identity/v3/users/596965ec955a4b989dc27603813d633c => generated 312 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:45.741286 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acce440e-0ad5-440a-ad5c-bdb66a91aaad None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:45.847149 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acce440e-0ad5-440a-ad5c-bdb66a91aaad None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:45.880618 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74346377-a89f-4f58-9169-61eb4e52eb24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:45.881220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74346377-a89f-4f58-9169-61eb4e52eb24 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:45.881494 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74346377-a89f-4f58-9169-61eb4e52eb24 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:45.881761 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74346377-a89f-4f58-9169-61eb4e52eb24 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:45.942195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:45.952006 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74346377-a89f-4f58-9169-61eb4e52eb24 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:45.954253 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74346377-a89f-4f58-9169-61eb4e52eb24 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:46.058231 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 573/1183] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:35:45 2022] GET /identity/v3/auth/tokens => generated 2123 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 15:35:46.065159 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39670681-00af-437b-9aa4-ac800beef98f None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:46.165352 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39670681-00af-437b-9aa4-ac800beef98f None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:46.200756 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38321cdc-641a-4ba4-8eda-4ad2ade29bb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:46.201399 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38321cdc-641a-4ba4-8eda-4ad2ade29bb3 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:46.201711 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38321cdc-641a-4ba4-8eda-4ad2ade29bb3 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:46.201988 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38321cdc-641a-4ba4-8eda-4ad2ade29bb3 None admin] PATH_INFO: `/v3/projects/d389a1e6c45c4ebc8a40f6722dd1604a` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:46.203285 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38321cdc-641a-4ba4-8eda-4ad2ade29bb3 None admin] RBAC: Authorizing `identity:get_project(project_id=d389a1e6c45c4ebc8a40f6722dd1604a)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:46.204762 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38321cdc-641a-4ba4-8eda-4ad2ade29bb3 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:46.206410 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 611/1184] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:35:46 2022] GET /identity/v3/projects/d389a1e6c45c4ebc8a40f6722dd1604a => generated 362 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:46.212547 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74346377-a89f-4f58-9169-61eb4e52eb24 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:46.307647 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:46.314315 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74346377-a89f-4f58-9169-61eb4e52eb24 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:46.348581 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c1ffa8-f57d-443b-9369-287f3e83e163 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:46.349174 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c1ffa8-f57d-443b-9369-287f3e83e163 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:46.349451 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c1ffa8-f57d-443b-9369-287f3e83e163 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:46.349716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c1ffa8-f57d-443b-9369-287f3e83e163 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:46.350473 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c1ffa8-f57d-443b-9369-287f3e83e163 None admin] RBAC: Adding query filter params (role.id=4d8fbbaea81a4145b89af12350bb067c, scope.project.id=d389a1e6c45c4ebc8a40f6722dd1604a, user.id=596965ec955a4b989dc27603813d633c) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:35:46.350958 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c1ffa8-f57d-443b-9369-287f3e83e163 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:46.352464 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c1ffa8-f57d-443b-9369-287f3e83e163 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:46.359267 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 574/1185] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:35:46 2022] GET /identity/v3/role_assignments?user.id=596965ec955a4b989dc27603813d633c&scope.project.id=d389a1e6c45c4ebc8a40f6722dd1604a&role.id=4d8fbbaea81a4145b89af12350bb067c => generated 260 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:48.275396 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a4ae63-6466-4410-a1ee-87d2eaefb56e None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:48.276023 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a4ae63-6466-4410-a1ee-87d2eaefb56e None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:48.276023 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a4ae63-6466-4410-a1ee-87d2eaefb56e None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:48.277319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 612/1186] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:35:48 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:35:48.283770 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8b4d98-80c0-4f63-9b92-198f2896ee10 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:48.284257 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8b4d98-80c0-4f63-9b92-198f2896ee10 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:48.284363 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8b4d98-80c0-4f63-9b92-198f2896ee10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:48.302568 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-bc8b4d98-80c0-4f63-9b92-198f2896ee10 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:35:48.303779 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-bc8b4d98-80c0-4f63-9b92-198f2896ee10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:35:48.368791 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 575/1187] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:35:48 2022] POST /identity/v3/auth/tokens => generated 2123 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:35:48.636662 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a4ae63-6466-4410-a1ee-87d2eaefb56e None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:48.651373 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:48.742719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a4ae63-6466-4410-a1ee-87d2eaefb56e None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:48.776557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee953ce-1867-4758-8dea-384b84e5e234 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:48.777172 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee953ce-1867-4758-8dea-384b84e5e234 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:48.777456 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee953ce-1867-4758-8dea-384b84e5e234 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:48.777728 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee953ce-1867-4758-8dea-384b84e5e234 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:48.788445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee953ce-1867-4758-8dea-384b84e5e234 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:35:48.793030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-cee953ce-1867-4758-8dea-384b84e5e234 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:35:48.793030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:35:48.793030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:35:48.793030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:35:48.793030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:35:48.793030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:35:48.793030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:35:48.793030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:35:48.793030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:35:48.793030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:35:48.793030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:35:48.793030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:35:48.793030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:35:48.793030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:35:48.793030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:35:48.793030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:35:48.794319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:35:48.794319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:35:48.794319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:35:48.794319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:35:48.794319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:35:48.794319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:35:48.794319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:35:48.794319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:35:48.794319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:35:48.794319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:35:48.794319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:35:48.794319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:35:48.794319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:35:48.794319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:35:48.794319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:35:48.794319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:35:48.796551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:35:48.796551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:35:48.796551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:35:48.796551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:35:48.796551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:35:48.796551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:35:48.796551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:35:48.796551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:35:48.796551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:35:48.796551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:35:48.796551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:35:48.796551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:35:48.796551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 613/1188] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:35:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:35:48.803279 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8b4d98-80c0-4f63-9b92-198f2896ee10 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:48.894838 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:48.904367 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8b4d98-80c0-4f63-9b92-198f2896ee10 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:48.938890 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8cae208-e2c4-4643-98da-c392b4126739 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:48.939483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cae208-e2c4-4643-98da-c392b4126739 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:48.939789 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cae208-e2c4-4643-98da-c392b4126739 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:48.940066 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cae208-e2c4-4643-98da-c392b4126739 None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:48.940725 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8cae208-e2c4-4643-98da-c392b4126739 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:35:48.941024 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8cae208-e2c4-4643-98da-c392b4126739 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:48.942634 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8cae208-e2c4-4643-98da-c392b4126739 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:48.950697 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 576/1189] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:35:48 2022] GET /identity/v3/roles?name=admin => generated 341 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:35:48.956616 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee953ce-1867-4758-8dea-384b84e5e234 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:49.028323 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:49.059963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee953ce-1867-4758-8dea-384b84e5e234 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:49.097405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efec833-4800-4b4d-8bf7-58a0be9373e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:49.098039 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efec833-4800-4b4d-8bf7-58a0be9373e2 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:49.098321 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efec833-4800-4b4d-8bf7-58a0be9373e2 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:49.098604 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efec833-4800-4b4d-8bf7-58a0be9373e2 None admin] PATH_INFO: `/v3/users/596965ec955a4b989dc27603813d633c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:49.099765 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5efec833-4800-4b4d-8bf7-58a0be9373e2 None admin] RBAC: Authorizing `identity:get_user(user_id=596965ec955a4b989dc27603813d633c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:49.101212 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5efec833-4800-4b4d-8bf7-58a0be9373e2 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:49.102987 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 614/1190] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:35:48 2022] GET /identity/v3/users/596965ec955a4b989dc27603813d633c => generated 312 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:49.109607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8cae208-e2c4-4643-98da-c392b4126739 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:49.211122 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8cae208-e2c4-4643-98da-c392b4126739 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:49.245001 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f2b366-6173-45ed-a26b-f72a6d0df85e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:49.245001 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f2b366-6173-45ed-a26b-f72a6d0df85e None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:49.245318 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f2b366-6173-45ed-a26b-f72a6d0df85e None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:49.245539 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f2b366-6173-45ed-a26b-f72a6d0df85e None admin] PATH_INFO: `/v3/projects/d389a1e6c45c4ebc8a40f6722dd1604a` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:49.246659 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f2b366-6173-45ed-a26b-f72a6d0df85e None admin] RBAC: Authorizing `identity:get_project(project_id=d389a1e6c45c4ebc8a40f6722dd1604a)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:49.248351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f2b366-6173-45ed-a26b-f72a6d0df85e None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:49.249987 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 577/1191] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:35:49 2022] GET /identity/v3/projects/d389a1e6c45c4ebc8a40f6722dd1604a => generated 362 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:49.256266 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efec833-4800-4b4d-8bf7-58a0be9373e2 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:49.359072 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efec833-4800-4b4d-8bf7-58a0be9373e2 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:49.400389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40bae925-c92d-44f1-8c7f-4bd8fd9dc671 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:49.401050 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40bae925-c92d-44f1-8c7f-4bd8fd9dc671 None admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:49.401332 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40bae925-c92d-44f1-8c7f-4bd8fd9dc671 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:49.401604 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40bae925-c92d-44f1-8c7f-4bd8fd9dc671 None admin] PATH_INFO: `/v3/projects/d389a1e6c45c4ebc8a40f6722dd1604a/users/596965ec955a4b989dc27603813d633c/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:49.403487 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40bae925-c92d-44f1-8c7f-4bd8fd9dc671 None admin] RBAC: Authorizing `identity:create_grant(project_id=d389a1e6c45c4ebc8a40f6722dd1604a, user_id=596965ec955a4b989dc27603813d633c, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:49.405708 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40bae925-c92d-44f1-8c7f-4bd8fd9dc671 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:49.414539 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:49.419667 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 615/1192] 10.222.0.102 () {58 vars in 1444 bytes} [Wed Aug 24 15:35:49 2022] PUT /identity/v3/projects/d389a1e6c45c4ebc8a40f6722dd1604a/users/596965ec955a4b989dc27603813d633c/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:35:51.323208 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264e4ab2-cef7-4f01-a6c8-153dab738acb None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:51.323758 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264e4ab2-cef7-4f01-a6c8-153dab738acb None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:51.323849 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264e4ab2-cef7-4f01-a6c8-153dab738acb None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:51.325112 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 578/1193] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:35:51 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:35:51.330973 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6137cc65-1611-4ee1-ad7e-7b02eba70d08 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:51.331368 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6137cc65-1611-4ee1-ad7e-7b02eba70d08 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:51.331527 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6137cc65-1611-4ee1-ad7e-7b02eba70d08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:51.349703 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-6137cc65-1611-4ee1-ad7e-7b02eba70d08 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:35:51.350918 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-6137cc65-1611-4ee1-ad7e-7b02eba70d08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:35:51.422828 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 616/1194] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:35:51 2022] POST /identity/v3/auth/tokens => generated 2123 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:35:51.704755 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264e4ab2-cef7-4f01-a6c8-153dab738acb None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:51.727491 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:51.811532 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264e4ab2-cef7-4f01-a6c8-153dab738acb None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:51.846586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099ef557-e3e0-48d6-8fd8-4a58b7adb339 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:51.847210 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099ef557-e3e0-48d6-8fd8-4a58b7adb339 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:51.847490 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099ef557-e3e0-48d6-8fd8-4a58b7adb339 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:51.847790 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099ef557-e3e0-48d6-8fd8-4a58b7adb339 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:51.858098 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-099ef557-e3e0-48d6-8fd8-4a58b7adb339 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:35:51.862731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-099ef557-e3e0-48d6-8fd8-4a58b7adb339 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:35:51.862731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:35:51.862731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:35:51.862731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:35:51.862731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:35:51.862731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:35:51.862731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:35:51.862731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:35:51.862731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:35:51.862731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:35:51.862731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:35:51.862731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:35:51.862731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:35:51.862731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:35:51.862731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:35:51.862731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:35:51.863847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:35:51.863847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:35:51.863847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:35:51.863847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:35:51.863847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:35:51.863847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:35:51.863847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:35:51.863847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:35:51.863847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:35:51.863847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:35:51.863847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:35:51.863847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:35:51.863847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:35:51.863847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:35:51.863847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:35:51.863847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:35:51.864683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:35:51.864683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:35:51.864683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:35:51.864683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:35:51.864683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:35:51.864683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:35:51.864683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:35:51.864683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:35:51.864683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:35:51.864683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:35:51.864683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:35:51.864683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:35:51.864683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 579/1195] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:35:51 2022] GET /identity/v3/roles/admin => generated 83 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:35:51.869924 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6137cc65-1611-4ee1-ad7e-7b02eba70d08 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:51.967968 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6137cc65-1611-4ee1-ad7e-7b02eba70d08 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:51.995265 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:52.001704 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f2d0ea-c327-4d13-8d60-ee3985476c80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:52.002311 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f2d0ea-c327-4d13-8d60-ee3985476c80 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:52.002594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f2d0ea-c327-4d13-8d60-ee3985476c80 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:52.002868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f2d0ea-c327-4d13-8d60-ee3985476c80 None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:52.003537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f2d0ea-c327-4d13-8d60-ee3985476c80 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:35:52.003876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f2d0ea-c327-4d13-8d60-ee3985476c80 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:52.005242 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f2d0ea-c327-4d13-8d60-ee3985476c80 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:52.013999 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 617/1196] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:35:51 2022] GET /identity/v3/roles?name=admin => generated 341 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:52.019983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099ef557-e3e0-48d6-8fd8-4a58b7adb339 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:52.104968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:52.125774 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099ef557-e3e0-48d6-8fd8-4a58b7adb339 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:52.159766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566e6d9c-de3c-4574-9013-5947fe3ed7b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:52.160446 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566e6d9c-de3c-4574-9013-5947fe3ed7b3 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:52.160749 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566e6d9c-de3c-4574-9013-5947fe3ed7b3 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:52.161050 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566e6d9c-de3c-4574-9013-5947fe3ed7b3 None admin] PATH_INFO: `/v3/users/596965ec955a4b989dc27603813d633c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:52.162237 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-566e6d9c-de3c-4574-9013-5947fe3ed7b3 None admin] RBAC: Authorizing `identity:get_user(user_id=596965ec955a4b989dc27603813d633c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:52.163784 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-566e6d9c-de3c-4574-9013-5947fe3ed7b3 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:52.165386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 580/1197] 10.222.0.102 () {56 vars in 1182 bytes} [Wed Aug 24 15:35:52 2022] GET /identity/v3/users/596965ec955a4b989dc27603813d633c => generated 312 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:52.172458 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f2d0ea-c327-4d13-8d60-ee3985476c80 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:52.271431 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f2d0ea-c327-4d13-8d60-ee3985476c80 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:52.307232 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d4810b-35c3-4efe-96d3-f4738ec9ee06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:52.307855 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d4810b-35c3-4efe-96d3-f4738ec9ee06 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:52.308136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d4810b-35c3-4efe-96d3-f4738ec9ee06 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:52.308408 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d4810b-35c3-4efe-96d3-f4738ec9ee06 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:52.365856 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:52.376633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d4810b-35c3-4efe-96d3-f4738ec9ee06 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:52.378096 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d4810b-35c3-4efe-96d3-f4738ec9ee06 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:52.485662 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 618/1198] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:35:52 2022] GET /identity/v3/auth/tokens => generated 2123 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 15:35:52.491780 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566e6d9c-de3c-4574-9013-5947fe3ed7b3 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:52.591890 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566e6d9c-de3c-4574-9013-5947fe3ed7b3 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:52.627327 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279af589-b87a-4c7e-830c-bb20b18ccce3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:52.627984 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279af589-b87a-4c7e-830c-bb20b18ccce3 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:52.628265 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279af589-b87a-4c7e-830c-bb20b18ccce3 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:52.628537 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279af589-b87a-4c7e-830c-bb20b18ccce3 None admin] PATH_INFO: `/v3/projects/d389a1e6c45c4ebc8a40f6722dd1604a` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:52.629668 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-279af589-b87a-4c7e-830c-bb20b18ccce3 None admin] RBAC: Authorizing `identity:get_project(project_id=d389a1e6c45c4ebc8a40f6722dd1604a)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:52.631101 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-279af589-b87a-4c7e-830c-bb20b18ccce3 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:52.632770 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 581/1199] 10.222.0.102 () {56 vars in 1191 bytes} [Wed Aug 24 15:35:52 2022] GET /identity/v3/projects/d389a1e6c45c4ebc8a40f6722dd1604a => generated 362 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:35:52.641689 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d4810b-35c3-4efe-96d3-f4738ec9ee06 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:35:52.747559 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:35:52.757834 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d4810b-35c3-4efe-96d3-f4738ec9ee06 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:35:52.791809 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fc820fc-2366-4c6e-835f-b09bd77a2051 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:35:52.792410 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc820fc-2366-4c6e-835f-b09bd77a2051 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:35:52.792695 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc820fc-2366-4c6e-835f-b09bd77a2051 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:35:52.793001 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc820fc-2366-4c6e-835f-b09bd77a2051 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:35:52.793726 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc820fc-2366-4c6e-835f-b09bd77a2051 None admin] RBAC: Adding query filter params (role.id=4d8fbbaea81a4145b89af12350bb067c, scope.project.id=d389a1e6c45c4ebc8a40f6722dd1604a, user.id=596965ec955a4b989dc27603813d633c) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:35:52.794652 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc820fc-2366-4c6e-835f-b09bd77a2051 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:35:52.796323 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc820fc-2366-4c6e-835f-b09bd77a2051 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:35:52.802112 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 619/1200] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:35:52 2022] GET /identity/v3/role_assignments?user.id=596965ec955a4b989dc27603813d633c&scope.project.id=d389a1e6c45c4ebc8a40f6722dd1604a&role.id=4d8fbbaea81a4145b89af12350bb067c => generated 610 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:36:02.623017 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=92 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:36:02.799517 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=22 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:36:50.009677 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd72a9c-06eb-4012-b807-17332694000c None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:36:50.010316 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd72a9c-06eb-4012-b807-17332694000c None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:36:50.010316 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd72a9c-06eb-4012-b807-17332694000c None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:36:50.011764 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 582/1201] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:36:50 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:36:50.019716 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dcea15-d66a-4bae-aeaa-03fe7656959e None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:36:50.019716 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dcea15-d66a-4bae-aeaa-03fe7656959e None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:36:50.023397 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dcea15-d66a-4bae-aeaa-03fe7656959e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:36:50.049906 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-74dcea15-d66a-4bae-aeaa-03fe7656959e None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:36:50.051431 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-74dcea15-d66a-4bae-aeaa-03fe7656959e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:36:50.147957 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 620/1202] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:36:50 2022] POST /identity/v3/auth/tokens => generated 2123 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:36:50.419691 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd72a9c-06eb-4012-b807-17332694000c None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:36:50.552715 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd72a9c-06eb-4012-b807-17332694000c None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:36:50.603829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48c154d-08a6-4871-9d8e-a8256f5ba1ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:36:50.604469 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48c154d-08a6-4871-9d8e-a8256f5ba1ad None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:36:50.604744 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48c154d-08a6-4871-9d8e-a8256f5ba1ad None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:36:50.605010 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48c154d-08a6-4871-9d8e-a8256f5ba1ad None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:36:50.618166 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48c154d-08a6-4871-9d8e-a8256f5ba1ad None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:36:50.619745 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48c154d-08a6-4871-9d8e-a8256f5ba1ad None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:36:50.638485 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-d48c154d-08a6-4871-9d8e-a8256f5ba1ad None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:36:50.638485 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:36:50.638485 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:36:50.638485 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:36:50.638485 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:36:50.638485 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:36:50.638485 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:36:50.638485 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:36:50.638485 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:36:50.638485 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:36:50.638485 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:36:50.638485 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:36:50.638485 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:36:50.638485 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:36:50.638485 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:36:50.638485 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:36:50.639621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:36:50.639621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:36:50.639621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:36:50.639621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:36:50.639621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:36:50.639621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:36:50.639621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:36:50.639621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:36:50.639621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:36:50.639621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:36:50.639621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:36:50.639621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:36:50.639621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:36:50.639621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:36:50.639621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:36:50.639621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:36:50.639621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:36:50.640626 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:36:50.640626 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:36:50.640626 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:36:50.640626 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 583/1203] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:36:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:36:50.646358 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74dcea15-d66a-4bae-aeaa-03fe7656959e None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:36:50.777730 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74dcea15-d66a-4bae-aeaa-03fe7656959e None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:36:50.831490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:36:50.839276 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ac065c-765e-4d3a-a17a-b1968b451e2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:36:50.839992 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ac065c-765e-4d3a-a17a-b1968b451e2c None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:36:50.840274 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ac065c-765e-4d3a-a17a-b1968b451e2c None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:36:50.840547 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ac065c-765e-4d3a-a17a-b1968b451e2c None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:36:50.841705 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ac065c-765e-4d3a-a17a-b1968b451e2c None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:36:50.842016 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ac065c-765e-4d3a-a17a-b1968b451e2c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:36:50.843556 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ac065c-765e-4d3a-a17a-b1968b451e2c None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:36:50.858122 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 621/1204] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:36:50 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:36:50.865555 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48c154d-08a6-4871-9d8e-a8256f5ba1ad None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:36:50.938778 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:36:51.003039 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48c154d-08a6-4871-9d8e-a8256f5ba1ad None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:36:51.049427 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3d56c1-723b-4648-bc66-48fd5e0e5b6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:36:51.050648 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3d56c1-723b-4648-bc66-48fd5e0e5b6d None admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:36:51.050902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3d56c1-723b-4648-bc66-48fd5e0e5b6d None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:36:51.051182 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3d56c1-723b-4648-bc66-48fd5e0e5b6d None admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:36:51.051897 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c3d56c1-723b-4648-bc66-48fd5e0e5b6d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:36:51.053350 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c3d56c1-723b-4648-bc66-48fd5e0e5b6d None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:36:51.079271 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 584/1205] 10.222.0.102 () {60 vars in 1137 bytes} [Wed Aug 24 15:36:50 2022] POST /identity/v3/users => generated 252 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 15:36:53.201279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69606dd4-e645-4697-b676-bc77b246d2fd None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:36:53.201810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69606dd4-e645-4697-b676-bc77b246d2fd None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:36:53.201810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69606dd4-e645-4697-b676-bc77b246d2fd None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:36:53.203183 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 622/1206] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:36:53 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:36:53.209464 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99837959-086c-46e4-a532-294e29f6d8ce None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:36:53.209970 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99837959-086c-46e4-a532-294e29f6d8ce None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:36:53.210093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99837959-086c-46e4-a532-294e29f6d8ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:36:53.229196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-99837959-086c-46e4-a532-294e29f6d8ce None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:36:53.230496 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-99837959-086c-46e4-a532-294e29f6d8ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:36:53.311802 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 585/1207] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:36:53 2022] POST /identity/v3/auth/tokens => generated 2123 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:36:53.597929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69606dd4-e645-4697-b676-bc77b246d2fd None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:36:53.746528 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69606dd4-e645-4697-b676-bc77b246d2fd None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:36:53.789543 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d25513-1cab-4dd3-af4f-5ebf4d676ff8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:36:53.790991 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d25513-1cab-4dd3-af4f-5ebf4d676ff8 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:36:53.790991 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d25513-1cab-4dd3-af4f-5ebf4d676ff8 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:36:53.790991 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d25513-1cab-4dd3-af4f-5ebf4d676ff8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:36:53.805104 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d25513-1cab-4dd3-af4f-5ebf4d676ff8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:36:53.810266 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-b6d25513-1cab-4dd3-af4f-5ebf4d676ff8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:36:53.810266 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:36:53.810266 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:36:53.810266 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:36:53.810266 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:36:53.810266 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:36:53.810266 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:36:53.810266 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:36:53.810266 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:36:53.810266 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:36:53.810266 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:36:53.810266 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:36:53.810266 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:36:53.810266 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:36:53.810266 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:36:53.810266 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:36:53.813011 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:36:53.813011 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:36:53.813011 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:36:53.813011 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:36:53.813011 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:36:53.813011 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:36:53.813011 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:36:53.813011 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:36:53.813011 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:36:53.813011 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:36:53.813011 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:36:53.813011 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:36:53.813011 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:36:53.813011 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:36:53.813011 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:36:53.813011 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:36:53.814547 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:36:53.814547 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:36:53.814547 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:36:53.814547 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:36:53.814547 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:36:53.814547 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:36:53.814547 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:36:53.814547 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:36:53.814547 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:36:53.814547 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:36:53.814547 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:36:53.814547 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:36:53.814547 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 623/1208] 10.222.0.102 () {56 vars in 1107 bytes} [Wed Aug 24 15:36:53 2022] GET /identity/v3/roles/service => generated 85 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:36:53.818174 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99837959-086c-46e4-a532-294e29f6d8ce None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:36:53.863498 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:36:53.936373 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99837959-086c-46e4-a532-294e29f6d8ce None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:36:53.982819 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc29be0-607e-4860-97ec-9a1bbe438004 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:36:53.983559 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc29be0-607e-4860-97ec-9a1bbe438004 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:36:53.983879 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc29be0-607e-4860-97ec-9a1bbe438004 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:36:53.984173 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc29be0-607e-4860-97ec-9a1bbe438004 None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:36:53.984857 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc29be0-607e-4860-97ec-9a1bbe438004 None admin] RBAC: Adding query filter params (name=service) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:36:53.985164 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc29be0-607e-4860-97ec-9a1bbe438004 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:36:53.987333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc29be0-607e-4860-97ec-9a1bbe438004 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:36:53.997566 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 586/1209] 10.222.0.102 () {56 vars in 1108 bytes} [Wed Aug 24 15:36:53 2022] GET /identity/v3/roles?name=service => generated 328 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:36:54.004227 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d25513-1cab-4dd3-af4f-5ebf4d676ff8 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:36:54.070649 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:36:54.131874 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d25513-1cab-4dd3-af4f-5ebf4d676ff8 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:36:54.177968 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57946399-ef65-4db1-b4b4-acfdcc1b849b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:36:54.178678 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57946399-ef65-4db1-b4b4-acfdcc1b849b None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:36:54.178919 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57946399-ef65-4db1-b4b4-acfdcc1b849b None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:36:54.179199 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57946399-ef65-4db1-b4b4-acfdcc1b849b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:36:54.194170 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57946399-ef65-4db1-b4b4-acfdcc1b849b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:36:54.195898 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57946399-ef65-4db1-b4b4-acfdcc1b849b None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:36:54.218541 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-57946399-ef65-4db1-b4b4-acfdcc1b849b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:36:54.218541 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:36:54.218541 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:36:54.218541 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:36:54.218541 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:36:54.218541 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:36:54.218541 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:36:54.218541 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:36:54.218541 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:36:54.218541 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:36:54.218541 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:36:54.218541 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:36:54.218541 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:36:54.218541 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:36:54.218541 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:36:54.218541 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:36:54.220963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:36:54.220963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:36:54.220963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:36:54.220963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:36:54.220963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:36:54.220963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:36:54.220963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:36:54.220963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:36:54.220963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:36:54.220963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:36:54.220963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:36:54.220963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:36:54.220963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:36:54.220963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:36:54.220963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:36:54.220963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:36:54.220963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:36:54.222020 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:36:54.222020 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:36:54.222020 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:36:54.222020 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 624/1210] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:36:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:36:54.227442 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc29be0-607e-4860-97ec-9a1bbe438004 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:36:54.350751 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:36:54.354339 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc29be0-607e-4860-97ec-9a1bbe438004 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:36:54.399072 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5329afbb-7e4a-4812-a4fc-6c8f27e14bfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:36:54.400207 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5329afbb-7e4a-4812-a4fc-6c8f27e14bfe None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:36:54.400207 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5329afbb-7e4a-4812-a4fc-6c8f27e14bfe None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:36:54.400509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5329afbb-7e4a-4812-a4fc-6c8f27e14bfe None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:36:54.401374 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5329afbb-7e4a-4812-a4fc-6c8f27e14bfe None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:36:54.401769 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5329afbb-7e4a-4812-a4fc-6c8f27e14bfe None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:36:54.403298 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5329afbb-7e4a-4812-a4fc-6c8f27e14bfe None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:36:54.419957 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 587/1211] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:36:54 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:36:54.425970 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57946399-ef65-4db1-b4b4-acfdcc1b849b None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:36:54.558202 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57946399-ef65-4db1-b4b4-acfdcc1b849b None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:36:54.560429 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:36:54.601669 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01714ead-f064-4da4-b070-0f8de4be7f50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:36:54.602377 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01714ead-f064-4da4-b070-0f8de4be7f50 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:36:54.602695 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01714ead-f064-4da4-b070-0f8de4be7f50 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:36:54.603036 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01714ead-f064-4da4-b070-0f8de4be7f50 None admin] PATH_INFO: `/v3/users/placement` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:36:54.612360 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01714ead-f064-4da4-b070-0f8de4be7f50 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:36:54.613972 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01714ead-f064-4da4-b070-0f8de4be7f50 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:36:54.626929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-01714ead-f064-4da4-b070-0f8de4be7f50 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 24 15:36:54.626929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:36:54.626929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:36:54.626929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:36:54.626929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:36:54.626929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:36:54.626929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:36:54.626929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:36:54.626929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:36:54.626929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:36:54.626929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:36:54.626929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:36:54.626929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:36:54.626929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:36:54.626929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:36:54.626929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:36:54.628503 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:36:54.628503 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:36:54.628503 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:36:54.628503 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:36:54.628503 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:36:54.628503 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:36:54.628503 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:36:54.628503 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:36:54.628503 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:36:54.628503 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:36:54.628503 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:36:54.628503 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:36:54.628503 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:36:54.628503 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:36:54.628503 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:36:54.628503 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:36:54.628503 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:36:54.628503 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:36:54.629963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:36:54.629963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:36:54.629963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:36:54.629963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:36:54.629963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:36:54.629963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:36:54.629963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:36:54.629963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:36:54.629963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 24 15:36:54.629963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:36:54.633230 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 625/1212] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:36:54 2022] GET /identity/v3/users/placement => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:36:54.635441 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5329afbb-7e4a-4812-a4fc-6c8f27e14bfe None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:36:54.757739 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5329afbb-7e4a-4812-a4fc-6c8f27e14bfe None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:36:54.795872 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2435e2-eaa3-4885-8336-08047bbc7023 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:36:54.796541 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2435e2-eaa3-4885-8336-08047bbc7023 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:36:54.796837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2435e2-eaa3-4885-8336-08047bbc7023 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:36:54.797110 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2435e2-eaa3-4885-8336-08047bbc7023 None admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:36:54.797953 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e2435e2-eaa3-4885-8336-08047bbc7023 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:36:54.798942 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e2435e2-eaa3-4885-8336-08047bbc7023 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:36:54.801202 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e2435e2-eaa3-4885-8336-08047bbc7023 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:36:54.835137 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 588/1213] 10.222.0.102 () {56 vars in 1148 bytes} [Wed Aug 24 15:36:54 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 380 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:36:54.836869 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01714ead-f064-4da4-b070-0f8de4be7f50 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:36:54.989541 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01714ead-f064-4da4-b070-0f8de4be7f50 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:36:55.041333 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16512781-2dcf-4c4c-b50d-c78b6c4ba21c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:36:55.041887 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16512781-2dcf-4c4c-b50d-c78b6c4ba21c None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:36:55.042153 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16512781-2dcf-4c4c-b50d-c78b6c4ba21c None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:36:55.042425 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16512781-2dcf-4c4c-b50d-c78b6c4ba21c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:36:55.080931 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:36:55.128652 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16512781-2dcf-4c4c-b50d-c78b6c4ba21c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:36:55.130107 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16512781-2dcf-4c4c-b50d-c78b6c4ba21c None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:36:55.240663 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 626/1214] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:36:54 2022] GET /identity/v3/auth/tokens => generated 2123 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 15:36:55.246835 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2435e2-eaa3-4885-8336-08047bbc7023 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:36:55.269741 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:36:55.378437 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2435e2-eaa3-4885-8336-08047bbc7023 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:36:55.418037 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddd74a6-ffe5-4869-b6c8-15c0923afb4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:36:55.418440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddd74a6-ffe5-4869-b6c8-15c0923afb4a None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:36:55.418526 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddd74a6-ffe5-4869-b6c8-15c0923afb4a None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:36:55.418801 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddd74a6-ffe5-4869-b6c8-15c0923afb4a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:36:55.434136 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ddd74a6-ffe5-4869-b6c8-15c0923afb4a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:36:55.435606 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ddd74a6-ffe5-4869-b6c8-15c0923afb4a None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:36:55.457401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-8ddd74a6-ffe5-4869-b6c8-15c0923afb4a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:36:55.457401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:36:55.457401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:36:55.457401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:36:55.457401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:36:55.457401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:36:55.457401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:36:55.457401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:36:55.457401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:36:55.457401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:36:55.457401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:36:55.457401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:36:55.457401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:36:55.457401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:36:55.457401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:36:55.457401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:36:55.458624 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:36:55.458624 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:36:55.458624 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:36:55.458624 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:36:55.458624 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:36:55.458624 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:36:55.458624 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:36:55.458624 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:36:55.458624 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:36:55.458624 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:36:55.458624 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:36:55.458624 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:36:55.458624 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:36:55.458624 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:36:55.458624 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:36:55.458624 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:36:55.458624 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:36:55.460025 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:36:55.460025 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:36:55.460025 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:36:55.460419 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 589/1215] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:36:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:36:55.465589 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16512781-2dcf-4c4c-b50d-c78b6c4ba21c None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:36:55.533584 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:36:55.577861 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16512781-2dcf-4c4c-b50d-c78b6c4ba21c None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:36:55.617242 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f1cb75-6cc6-47ed-bcf1-207f9d4a4831 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:36:55.617868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f1cb75-6cc6-47ed-bcf1-207f9d4a4831 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:36:55.618148 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f1cb75-6cc6-47ed-bcf1-207f9d4a4831 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:36:55.618148 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f1cb75-6cc6-47ed-bcf1-207f9d4a4831 None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:36:55.618148 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f1cb75-6cc6-47ed-bcf1-207f9d4a4831 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:36:55.618148 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f1cb75-6cc6-47ed-bcf1-207f9d4a4831 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:36:55.621059 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f1cb75-6cc6-47ed-bcf1-207f9d4a4831 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:36:55.637617 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 627/1216] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:36:55 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:36:55.644159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddd74a6-ffe5-4869-b6c8-15c0923afb4a None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:36:55.729725 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:36:55.780846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddd74a6-ffe5-4869-b6c8-15c0923afb4a None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:36:55.840144 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7226b81f-594b-4586-80db-87a3ae25e1ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:36:55.840803 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7226b81f-594b-4586-80db-87a3ae25e1ef None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:36:55.846268 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7226b81f-594b-4586-80db-87a3ae25e1ef None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:36:55.846268 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7226b81f-594b-4586-80db-87a3ae25e1ef None admin] PATH_INFO: `/v3/projects/service` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:36:55.853983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7226b81f-594b-4586-80db-87a3ae25e1ef None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:36:55.855542 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7226b81f-594b-4586-80db-87a3ae25e1ef None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:36:55.871434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-7226b81f-594b-4586-80db-87a3ae25e1ef None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:36:55.871434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:36:55.871434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:36:55.871434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:36:55.871434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:36:55.871434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:36:55.871434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:36:55.871434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:36:55.871434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:36:55.871434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:36:55.871434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:36:55.871434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:36:55.871434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:36:55.871434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:36:55.871434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:36:55.872654 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:36:55.872654 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:36:55.872654 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:36:55.872654 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:36:55.872654 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:36:55.872654 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:36:55.872654 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:36:55.872654 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:36:55.872654 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:36:55.872654 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:36:55.872654 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:36:55.872654 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:36:55.872654 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:36:55.872654 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:36:55.872654 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:36:55.872654 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:36:55.872654 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:36:55.873776 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:36:55.873776 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:36:55.873776 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:36:55.873776 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:36:55.873776 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:36:55.873776 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:36:55.873776 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:36:55.873776 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:36:55.875093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 590/1217] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:36:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:36:55.881461 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f1cb75-6cc6-47ed-bcf1-207f9d4a4831 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:36:56.022398 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f1cb75-6cc6-47ed-bcf1-207f9d4a4831 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:36:56.061149 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:36:56.075485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42726fa-d43f-46ee-9c4e-29c5b96fe5b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:36:56.076254 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42726fa-d43f-46ee-9c4e-29c5b96fe5b3 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:36:56.076600 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42726fa-d43f-46ee-9c4e-29c5b96fe5b3 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:36:56.076901 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42726fa-d43f-46ee-9c4e-29c5b96fe5b3 None admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:36:56.077624 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42726fa-d43f-46ee-9c4e-29c5b96fe5b3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:36:56.077920 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42726fa-d43f-46ee-9c4e-29c5b96fe5b3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:36:56.079785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42726fa-d43f-46ee-9c4e-29c5b96fe5b3 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:36:56.099234 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 628/1218] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:36:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:36:56.105224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7226b81f-594b-4586-80db-87a3ae25e1ef None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:36:56.230960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7226b81f-594b-4586-80db-87a3ae25e1ef None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:36:56.262652 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:36:56.272481 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a509e6cd-d66c-427e-a907-84e049956596 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:36:56.273159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a509e6cd-d66c-427e-a907-84e049956596 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:36:56.273448 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a509e6cd-d66c-427e-a907-84e049956596 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:36:56.273812 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a509e6cd-d66c-427e-a907-84e049956596 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:36:56.274604 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a509e6cd-d66c-427e-a907-84e049956596 None admin] RBAC: Adding query filter params (role.id=628bab09d1dc48fd9e01023b2f7671f6, scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5, user.id=0ee0002a76c3471f819b37fef510ef99) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:36:56.274984 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a509e6cd-d66c-427e-a907-84e049956596 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:36:56.276534 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a509e6cd-d66c-427e-a907-84e049956596 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:36:56.296079 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 591/1219] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:36:56 2022] GET /identity/v3/role_assignments?user.id=0ee0002a76c3471f819b37fef510ef99&scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5&role.id=628bab09d1dc48fd9e01023b2f7671f6 => generated 260 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:36:58.342790 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a90424-6837-49d3-86ac-ebc627e6a79f None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:36:58.343418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a90424-6837-49d3-86ac-ebc627e6a79f None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:36:58.343418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a90424-6837-49d3-86ac-ebc627e6a79f None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:36:58.345009 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 629/1220] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:36:58 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:36:58.354815 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2859be7-d1f4-4418-9d35-9204abf5cbbd None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:36:58.355386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2859be7-d1f4-4418-9d35-9204abf5cbbd None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:36:58.355386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2859be7-d1f4-4418-9d35-9204abf5cbbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:36:58.374663 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-f2859be7-d1f4-4418-9d35-9204abf5cbbd None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:36:58.376853 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-f2859be7-d1f4-4418-9d35-9204abf5cbbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:36:58.465232 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 592/1221] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:36:58 2022] POST /identity/v3/auth/tokens => generated 2123 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:36:58.739764 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a90424-6837-49d3-86ac-ebc627e6a79f None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:36:58.859408 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a90424-6837-49d3-86ac-ebc627e6a79f None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:36:58.898026 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c1b895-94d0-4a31-9ac4-faa8441c26f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:36:58.898988 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c1b895-94d0-4a31-9ac4-faa8441c26f6 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:36:58.899260 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c1b895-94d0-4a31-9ac4-faa8441c26f6 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:36:58.899527 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c1b895-94d0-4a31-9ac4-faa8441c26f6 None admin] PATH_INFO: `/v3/roles/service` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:36:58.919394 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c1b895-94d0-4a31-9ac4-faa8441c26f6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:36:58.925083 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-a7c1b895-94d0-4a31-9ac4-faa8441c26f6 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:36:58.925083 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:36:58.925083 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:36:58.925083 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:36:58.925083 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:36:58.925083 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:36:58.925083 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:36:58.925083 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:36:58.925083 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:36:58.925083 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:36:58.925083 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:36:58.925083 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:36:58.925083 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:36:58.925083 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:36:58.925083 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:36:58.925083 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:36:58.926206 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:36:58.926206 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:36:58.926206 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:36:58.926206 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:36:58.926206 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:36:58.926206 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:36:58.926206 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:36:58.926206 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:36:58.926206 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:36:58.926206 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:36:58.926206 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:36:58.926206 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:36:58.926206 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:36:58.926206 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:36:58.926206 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:36:58.926206 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:36:58.927100 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:36:58.927100 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:36:58.927100 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:36:58.927100 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:36:58.927100 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:36:58.927100 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:36:58.927100 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:36:58.927100 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:36:58.927100 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:36:58.927100 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:36:58.927100 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:36:58.927100 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:36:58.927100 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 630/1222] 10.222.0.102 () {56 vars in 1107 bytes} [Wed Aug 24 15:36:58 2022] GET /identity/v3/roles/service => generated 85 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:36:58.932747 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2859be7-d1f4-4418-9d35-9204abf5cbbd None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:36:59.053084 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2859be7-d1f4-4418-9d35-9204abf5cbbd None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:36:59.080808 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:36:59.092765 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd1a2d6-69de-4585-8e8e-57a24e008d30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:36:59.093373 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd1a2d6-69de-4585-8e8e-57a24e008d30 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:36:59.093665 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd1a2d6-69de-4585-8e8e-57a24e008d30 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:36:59.093925 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd1a2d6-69de-4585-8e8e-57a24e008d30 None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:36:59.094653 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd1a2d6-69de-4585-8e8e-57a24e008d30 None admin] RBAC: Adding query filter params (name=service) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:36:59.095168 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd1a2d6-69de-4585-8e8e-57a24e008d30 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:36:59.096593 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd1a2d6-69de-4585-8e8e-57a24e008d30 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:36:59.110509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 593/1223] 10.222.0.102 () {56 vars in 1108 bytes} [Wed Aug 24 15:36:58 2022] GET /identity/v3/roles?name=service => generated 328 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:36:59.111857 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c1b895-94d0-4a31-9ac4-faa8441c26f6 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:36:59.165438 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:36:59.230743 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c1b895-94d0-4a31-9ac4-faa8441c26f6 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:36:59.280361 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36205c55-1ffb-4db3-9c02-016a1493617a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:36:59.281435 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36205c55-1ffb-4db3-9c02-016a1493617a None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:36:59.281845 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36205c55-1ffb-4db3-9c02-016a1493617a None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:36:59.282239 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36205c55-1ffb-4db3-9c02-016a1493617a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:36:59.298074 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36205c55-1ffb-4db3-9c02-016a1493617a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:36:59.300653 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36205c55-1ffb-4db3-9c02-016a1493617a None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:36:59.326699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-36205c55-1ffb-4db3-9c02-016a1493617a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:36:59.326699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:36:59.326699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:36:59.326699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:36:59.326699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:36:59.326699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:36:59.326699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:36:59.326699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:36:59.326699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:36:59.326699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:36:59.326699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:36:59.326699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:36:59.326699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:36:59.326699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:36:59.326699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:36:59.326699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:36:59.327895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:36:59.327895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:36:59.327895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:36:59.327895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:36:59.327895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:36:59.327895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:36:59.327895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:36:59.327895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:36:59.327895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:36:59.327895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:36:59.327895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:36:59.327895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:36:59.327895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:36:59.327895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:36:59.327895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:36:59.327895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:36:59.327895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:36:59.328996 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:36:59.328996 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:36:59.328996 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:36:59.331789 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 631/1224] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:36:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:36:59.337310 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd1a2d6-69de-4585-8e8e-57a24e008d30 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:36:59.455467 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd1a2d6-69de-4585-8e8e-57a24e008d30 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:36:59.493553 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19810e4-c4ca-4147-a612-01320eafcc18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:36:59.494158 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19810e4-c4ca-4147-a612-01320eafcc18 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:36:59.494459 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19810e4-c4ca-4147-a612-01320eafcc18 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:36:59.494709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19810e4-c4ca-4147-a612-01320eafcc18 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:36:59.495350 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d19810e4-c4ca-4147-a612-01320eafcc18 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:36:59.495674 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d19810e4-c4ca-4147-a612-01320eafcc18 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:36:59.497049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d19810e4-c4ca-4147-a612-01320eafcc18 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:36:59.513410 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 594/1225] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:36:59 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:36:59.519616 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36205c55-1ffb-4db3-9c02-016a1493617a None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:36:59.639723 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:36:59.649549 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36205c55-1ffb-4db3-9c02-016a1493617a None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:36:59.690727 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d842b7-fa97-4fa3-bfcc-32e4422802fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:36:59.691496 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d842b7-fa97-4fa3-bfcc-32e4422802fb None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:36:59.691807 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d842b7-fa97-4fa3-bfcc-32e4422802fb None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:36:59.692080 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d842b7-fa97-4fa3-bfcc-32e4422802fb None admin] PATH_INFO: `/v3/users/placement` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:36:59.700767 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d842b7-fa97-4fa3-bfcc-32e4422802fb None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:36:59.702297 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d842b7-fa97-4fa3-bfcc-32e4422802fb None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:36:59.717279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-25d842b7-fa97-4fa3-bfcc-32e4422802fb None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 24 15:36:59.717279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:36:59.717279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:36:59.717279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:36:59.717279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:36:59.717279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:36:59.717279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:36:59.717279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:36:59.717279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:36:59.717279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:36:59.717279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:36:59.717279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:36:59.717279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:36:59.717279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:36:59.717279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:36:59.717279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:36:59.718306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:36:59.718306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:36:59.718306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:36:59.718306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:36:59.718306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:36:59.718306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:36:59.718306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:36:59.718306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:36:59.718306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:36:59.718306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:36:59.718306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:36:59.718306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:36:59.718306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:36:59.718306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:36:59.718306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:36:59.718306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:36:59.718306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:36:59.718306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:36:59.719829 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:36:59.719829 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:36:59.719829 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:36:59.719829 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:36:59.719829 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:36:59.719829 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:36:59.719829 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:36:59.719829 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:36:59.719829 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 24 15:36:59.719829 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:36:59.719829 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 632/1226] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:36:59 2022] GET /identity/v3/users/placement => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:36:59.727297 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19810e4-c4ca-4147-a612-01320eafcc18 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:36:59.795575 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:36:59.879294 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19810e4-c4ca-4147-a612-01320eafcc18 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:36:59.926525 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a07a16d-bdad-4296-8225-4c6242083f64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:36:59.927141 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a07a16d-bdad-4296-8225-4c6242083f64 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:36:59.927493 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a07a16d-bdad-4296-8225-4c6242083f64 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:36:59.927885 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a07a16d-bdad-4296-8225-4c6242083f64 None admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:36:59.928760 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a07a16d-bdad-4296-8225-4c6242083f64 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:36:59.929155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a07a16d-bdad-4296-8225-4c6242083f64 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:36:59.931753 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a07a16d-bdad-4296-8225-4c6242083f64 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:36:59.954836 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 595/1227] 10.222.0.102 () {56 vars in 1148 bytes} [Wed Aug 24 15:36:59 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 380 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:36:59.961361 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d842b7-fa97-4fa3-bfcc-32e4422802fb None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:00.091439 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d842b7-fa97-4fa3-bfcc-32e4422802fb None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:00.143278 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52462f56-fd55-4f74-a55f-f5baf13c63c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:00.144052 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52462f56-fd55-4f74-a55f-f5baf13c63c4 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:00.144386 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52462f56-fd55-4f74-a55f-f5baf13c63c4 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:00.144683 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52462f56-fd55-4f74-a55f-f5baf13c63c4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:00.159137 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52462f56-fd55-4f74-a55f-f5baf13c63c4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:00.160780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52462f56-fd55-4f74-a55f-f5baf13c63c4 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:00.180669 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-52462f56-fd55-4f74-a55f-f5baf13c63c4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:37:00.180669 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:37:00.180669 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:37:00.180669 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:37:00.180669 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:37:00.180669 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:37:00.180669 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:37:00.180669 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:37:00.180669 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:37:00.180669 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:37:00.180669 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:37:00.180669 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:37:00.180669 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:37:00.180669 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:37:00.180669 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:37:00.180669 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:37:00.181744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:37:00.181744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:37:00.181744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:37:00.181744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:37:00.181744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:37:00.181744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:37:00.181744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:37:00.181744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:37:00.181744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:37:00.181744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:37:00.181744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:37:00.181744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:37:00.181744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:37:00.181744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:37:00.181744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:37:00.181744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:37:00.181744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:37:00.182696 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:37:00.182696 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:37:00.182696 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:37:00.182696 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 633/1228] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:36:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:37:00.188549 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a07a16d-bdad-4296-8225-4c6242083f64 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:00.316392 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a07a16d-bdad-4296-8225-4c6242083f64 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:00.319303 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:00.360266 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f99e0c-945a-4e4c-b343-acc2fa4d63e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:00.360879 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f99e0c-945a-4e4c-b343-acc2fa4d63e5 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:00.361157 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f99e0c-945a-4e4c-b343-acc2fa4d63e5 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:00.361423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f99e0c-945a-4e4c-b343-acc2fa4d63e5 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:00.362077 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f99e0c-945a-4e4c-b343-acc2fa4d63e5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:37:00.362475 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f99e0c-945a-4e4c-b343-acc2fa4d63e5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:00.364025 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f99e0c-945a-4e4c-b343-acc2fa4d63e5 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:00.379211 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 596/1229] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:37:00 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:37:00.385462 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52462f56-fd55-4f74-a55f-f5baf13c63c4 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:00.396222 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:00.514076 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52462f56-fd55-4f74-a55f-f5baf13c63c4 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:00.559473 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c59c4b9-66fa-461d-b4ed-6f211f3817b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:00.560341 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c59c4b9-66fa-461d-b4ed-6f211f3817b7 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:00.560441 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c59c4b9-66fa-461d-b4ed-6f211f3817b7 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:00.560750 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c59c4b9-66fa-461d-b4ed-6f211f3817b7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:00.569205 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c59c4b9-66fa-461d-b4ed-6f211f3817b7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:00.570737 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c59c4b9-66fa-461d-b4ed-6f211f3817b7 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:00.583395 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-2c59c4b9-66fa-461d-b4ed-6f211f3817b7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:37:00.583395 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:37:00.583395 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:37:00.583395 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:37:00.583395 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:37:00.583395 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:37:00.583395 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:37:00.583395 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:37:00.583395 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:37:00.583395 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:37:00.583395 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:37:00.583395 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:37:00.583395 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:37:00.583395 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:37:00.583395 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:37:00.584513 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:37:00.584513 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:37:00.584513 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:37:00.584513 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:37:00.584513 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:37:00.584513 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:37:00.584513 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:37:00.584513 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:37:00.584513 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:37:00.584513 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:37:00.584513 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:37:00.584513 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:37:00.584513 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:37:00.584513 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:37:00.584513 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:37:00.584513 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:37:00.584513 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:37:00.585652 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:37:00.585652 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:37:00.585652 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:37:00.585652 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:37:00.585652 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:37:00.585652 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:37:00.585652 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:37:00.585652 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:37:00.589786 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 634/1230] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:37:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:37:00.591019 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f99e0c-945a-4e4c-b343-acc2fa4d63e5 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:00.708477 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f99e0c-945a-4e4c-b343-acc2fa4d63e5 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:00.760913 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a075c0f2-c539-478f-bb4f-894a8ddccae9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:00.761739 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a075c0f2-c539-478f-bb4f-894a8ddccae9 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:00.761920 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a075c0f2-c539-478f-bb4f-894a8ddccae9 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:00.762281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a075c0f2-c539-478f-bb4f-894a8ddccae9 None admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:00.762795 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a075c0f2-c539-478f-bb4f-894a8ddccae9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:37:00.763141 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a075c0f2-c539-478f-bb4f-894a8ddccae9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:00.764753 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a075c0f2-c539-478f-bb4f-894a8ddccae9 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:00.780953 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 597/1231] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:37:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:37:00.787860 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c59c4b9-66fa-461d-b4ed-6f211f3817b7 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:00.887096 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:00.955570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c59c4b9-66fa-461d-b4ed-6f211f3817b7 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:00.994480 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72b9577-fca2-420f-843b-f7117781d6d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:00.995160 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72b9577-fca2-420f-843b-f7117781d6d5 None admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:00.995609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72b9577-fca2-420f-843b-f7117781d6d5 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:00.995992 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72b9577-fca2-420f-843b-f7117781d6d5 None admin] PATH_INFO: `/v3/projects/3e50e6f6c1f44c648337c8d37a8e94a5/users/0ee0002a76c3471f819b37fef510ef99/roles/628bab09d1dc48fd9e01023b2f7671f6` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:01.015075 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72b9577-fca2-420f-843b-f7117781d6d5 None admin] RBAC: Authorizing `identity:create_grant(project_id=3e50e6f6c1f44c648337c8d37a8e94a5, user_id=0ee0002a76c3471f819b37fef510ef99, role_id=628bab09d1dc48fd9e01023b2f7671f6)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:01.016586 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72b9577-fca2-420f-843b-f7117781d6d5 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:01.035162 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 635/1232] 10.222.0.102 () {58 vars in 1444 bytes} [Wed Aug 24 15:37:00 2022] PUT /identity/v3/projects/3e50e6f6c1f44c648337c8d37a8e94a5/users/0ee0002a76c3471f819b37fef510ef99/roles/628bab09d1dc48fd9e01023b2f7671f6 => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 15:37:03.281510 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee14710-6b67-4bb3-b444-517ddeae7d52 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:03.282202 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee14710-6b67-4bb3-b444-517ddeae7d52 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:03.282202 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee14710-6b67-4bb3-b444-517ddeae7d52 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:03.284482 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 598/1233] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:37:03 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:37:03.291741 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70971503-dcba-481c-b79a-39ac2f473189 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:03.292921 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70971503-dcba-481c-b79a-39ac2f473189 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:03.293378 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70971503-dcba-481c-b79a-39ac2f473189 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:03.315658 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-70971503-dcba-481c-b79a-39ac2f473189 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:37:03.316938 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-70971503-dcba-481c-b79a-39ac2f473189 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:37:03.386769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 636/1234] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:37:03 2022] POST /identity/v3/auth/tokens => generated 2123 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:37:03.674853 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee14710-6b67-4bb3-b444-517ddeae7d52 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:03.708655 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:03.807451 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee14710-6b67-4bb3-b444-517ddeae7d52 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:03.862008 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4428198-2c41-4e2f-bccd-af655290f7d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:03.862627 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4428198-2c41-4e2f-bccd-af655290f7d3 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:03.862800 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4428198-2c41-4e2f-bccd-af655290f7d3 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:03.863017 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4428198-2c41-4e2f-bccd-af655290f7d3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:03.877650 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4428198-2c41-4e2f-bccd-af655290f7d3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:37:03.882560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-c4428198-2c41-4e2f-bccd-af655290f7d3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:37:03.882560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:37:03.882560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:37:03.882560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:37:03.882560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:37:03.882560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:37:03.882560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:37:03.882560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:37:03.882560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:37:03.882560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:37:03.882560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:37:03.882560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:37:03.882560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:37:03.882560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:37:03.882560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:37:03.882560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:37:03.884800 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:37:03.884800 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:37:03.884800 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:37:03.884800 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:37:03.884800 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:37:03.884800 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:37:03.884800 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:37:03.884800 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:37:03.884800 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:37:03.884800 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:37:03.884800 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:37:03.884800 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:37:03.884800 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:37:03.884800 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:37:03.884800 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:37:03.884800 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:37:03.886244 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:37:03.886244 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:37:03.886244 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:37:03.886244 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:37:03.886244 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:37:03.886244 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:37:03.886244 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:37:03.886244 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:37:03.886244 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:37:03.886244 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:37:03.886244 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 15:37:03.886244 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:37:03.886244 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 599/1235] 10.222.0.102 () {56 vars in 1107 bytes} [Wed Aug 24 15:37:03 2022] GET /identity/v3/roles/service => generated 85 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:37:03.890581 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70971503-dcba-481c-b79a-39ac2f473189 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:03.959416 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:04.045112 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70971503-dcba-481c-b79a-39ac2f473189 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:04.095655 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58868f5-f75c-4ea5-98d0-612576a1f2aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:04.096425 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58868f5-f75c-4ea5-98d0-612576a1f2aa None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:04.096736 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58868f5-f75c-4ea5-98d0-612576a1f2aa None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:04.097044 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58868f5-f75c-4ea5-98d0-612576a1f2aa None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:04.097691 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58868f5-f75c-4ea5-98d0-612576a1f2aa None admin] RBAC: Adding query filter params (name=service) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:37:04.097992 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58868f5-f75c-4ea5-98d0-612576a1f2aa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:04.099372 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58868f5-f75c-4ea5-98d0-612576a1f2aa None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:04.110560 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 637/1236] 10.222.0.102 () {56 vars in 1108 bytes} [Wed Aug 24 15:37:03 2022] GET /identity/v3/roles?name=service => generated 328 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:37:04.116905 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4428198-2c41-4e2f-bccd-af655290f7d3 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:04.225695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:04.246341 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4428198-2c41-4e2f-bccd-af655290f7d3 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:04.303535 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3cb625-2442-4b15-8719-59f6ee91c98f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:04.304705 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3cb625-2442-4b15-8719-59f6ee91c98f None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:04.305030 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3cb625-2442-4b15-8719-59f6ee91c98f None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:04.305229 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3cb625-2442-4b15-8719-59f6ee91c98f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:04.325931 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3cb625-2442-4b15-8719-59f6ee91c98f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:04.327620 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3cb625-2442-4b15-8719-59f6ee91c98f None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:04.348370 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-dc3cb625-2442-4b15-8719-59f6ee91c98f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:37:04.348370 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:37:04.348370 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:37:04.348370 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:37:04.348370 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:37:04.348370 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:37:04.348370 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:37:04.348370 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:37:04.348370 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:37:04.348370 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:37:04.348370 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:37:04.348370 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:37:04.348370 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:37:04.348370 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:37:04.348370 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:37:04.348370 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:37:04.349582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:37:04.349582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:37:04.349582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:37:04.349582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:37:04.349582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:37:04.349582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:37:04.349582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:37:04.349582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:37:04.349582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:37:04.349582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:37:04.349582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:37:04.349582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:37:04.349582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:37:04.349582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:37:04.349582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:37:04.349582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:37:04.349582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:37:04.351024 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:37:04.351024 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:37:04.351024 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:37:04.351024 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 600/1237] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:37:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:37:04.355954 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58868f5-f75c-4ea5-98d0-612576a1f2aa None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:04.520642 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58868f5-f75c-4ea5-98d0-612576a1f2aa None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:04.538238 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:04.570698 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99db6931-2644-45dd-951b-85c4ec439c1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:04.571325 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99db6931-2644-45dd-951b-85c4ec439c1a None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:04.571607 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99db6931-2644-45dd-951b-85c4ec439c1a None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:04.571936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99db6931-2644-45dd-951b-85c4ec439c1a None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:04.572594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99db6931-2644-45dd-951b-85c4ec439c1a None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:37:04.572890 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99db6931-2644-45dd-951b-85c4ec439c1a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:04.574279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99db6931-2644-45dd-951b-85c4ec439c1a None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:04.592242 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 638/1238] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:37:04 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:37:04.598641 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3cb625-2442-4b15-8719-59f6ee91c98f None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:04.735175 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3cb625-2442-4b15-8719-59f6ee91c98f None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:04.782820 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274c92ba-e1b4-49cb-b728-a689691bc780 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:04.783370 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274c92ba-e1b4-49cb-b728-a689691bc780 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:04.783681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274c92ba-e1b4-49cb-b728-a689691bc780 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:04.783957 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274c92ba-e1b4-49cb-b728-a689691bc780 None admin] PATH_INFO: `/v3/users/placement` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:04.786836 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:04.793920 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-274c92ba-e1b4-49cb-b728-a689691bc780 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:04.795930 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-274c92ba-e1b4-49cb-b728-a689691bc780 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:04.809686 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-274c92ba-e1b4-49cb-b728-a689691bc780 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 24 15:37:04.809686 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:37:04.809686 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:37:04.809686 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:37:04.809686 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:37:04.809686 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:37:04.809686 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:37:04.809686 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:37:04.809686 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:37:04.809686 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:37:04.809686 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:37:04.809686 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:37:04.809686 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:37:04.809686 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:37:04.809686 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:37:04.809686 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:37:04.810918 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:37:04.810918 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:37:04.810918 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:37:04.810918 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:37:04.810918 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:37:04.810918 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:37:04.810918 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:37:04.810918 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:37:04.810918 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:37:04.810918 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:37:04.810918 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:37:04.810918 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:37:04.810918 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:37:04.810918 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:37:04.810918 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:37:04.810918 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:37:04.810918 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:37:04.810918 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:37:04.811892 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:37:04.811892 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:37:04.811892 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:37:04.811892 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:37:04.811892 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:37:04.811892 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:37:04.811892 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:37:04.811892 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:37:04.811892 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 24 15:37:04.811892 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:37:04.811892 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 601/1239] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:37:04 2022] GET /identity/v3/users/placement => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:37:04.817785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99db6931-2644-45dd-951b-85c4ec439c1a None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:04.937959 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99db6931-2644-45dd-951b-85c4ec439c1a None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:04.976921 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616ab0bb-f902-49a5-85a0-f43875047d4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:04.977531 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616ab0bb-f902-49a5-85a0-f43875047d4d None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:04.977813 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616ab0bb-f902-49a5-85a0-f43875047d4d None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:04.978125 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616ab0bb-f902-49a5-85a0-f43875047d4d None admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:04.978833 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616ab0bb-f902-49a5-85a0-f43875047d4d None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:37:04.979147 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616ab0bb-f902-49a5-85a0-f43875047d4d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:04.980640 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616ab0bb-f902-49a5-85a0-f43875047d4d None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:05.001100 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 639/1240] 10.222.0.102 () {56 vars in 1148 bytes} [Wed Aug 24 15:37:04 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 380 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:37:05.010261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274c92ba-e1b4-49cb-b728-a689691bc780 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:05.160960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274c92ba-e1b4-49cb-b728-a689691bc780 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:05.199584 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9d40d2-4fb6-4381-879a-897db5d59db3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:05.200224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9d40d2-4fb6-4381-879a-897db5d59db3 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:05.200513 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9d40d2-4fb6-4381-879a-897db5d59db3 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:05.200787 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9d40d2-4fb6-4381-879a-897db5d59db3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:05.271790 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:05.280418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9d40d2-4fb6-4381-879a-897db5d59db3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:05.282940 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9d40d2-4fb6-4381-879a-897db5d59db3 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:05.415269 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 602/1241] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:37:05 2022] GET /identity/v3/auth/tokens => generated 2123 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 15:37:05.416533 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616ab0bb-f902-49a5-85a0-f43875047d4d None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:05.449253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:05.543886 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616ab0bb-f902-49a5-85a0-f43875047d4d None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:05.590185 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb86809f-997a-4f96-87ef-eca65bc685c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:05.590820 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb86809f-997a-4f96-87ef-eca65bc685c1 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:05.591106 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb86809f-997a-4f96-87ef-eca65bc685c1 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:05.591381 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb86809f-997a-4f96-87ef-eca65bc685c1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:05.604079 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb86809f-997a-4f96-87ef-eca65bc685c1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:05.606129 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb86809f-997a-4f96-87ef-eca65bc685c1 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:05.623284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-eb86809f-997a-4f96-87ef-eca65bc685c1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:37:05.623284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:37:05.623284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:37:05.623284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:37:05.623284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:37:05.623284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:37:05.623284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:37:05.623284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:37:05.623284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:37:05.623284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:37:05.623284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:37:05.623284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:37:05.623284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:37:05.623284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:37:05.623284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:37:05.623284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:37:05.625106 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:37:05.625106 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:37:05.625106 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:37:05.625106 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:37:05.625106 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:37:05.625106 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:37:05.625106 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:37:05.625106 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:37:05.625106 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:37:05.625106 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:37:05.625106 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:37:05.625106 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:37:05.625106 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:37:05.625106 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:37:05.625106 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:37:05.625106 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:37:05.625106 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:37:05.627301 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:37:05.627301 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:37:05.627301 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:37:05.627301 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 640/1242] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:37:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:37:05.631258 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9d40d2-4fb6-4381-879a-897db5d59db3 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:05.748910 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:05.753365 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9d40d2-4fb6-4381-879a-897db5d59db3 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:05.791355 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954e3e43-d1e0-4728-9379-7cfa0df5769b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:05.791988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954e3e43-d1e0-4728-9379-7cfa0df5769b None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:05.792260 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954e3e43-d1e0-4728-9379-7cfa0df5769b None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:05.792530 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954e3e43-d1e0-4728-9379-7cfa0df5769b None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:05.793201 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954e3e43-d1e0-4728-9379-7cfa0df5769b None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:37:05.793472 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954e3e43-d1e0-4728-9379-7cfa0df5769b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:05.794832 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954e3e43-d1e0-4728-9379-7cfa0df5769b None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:05.809053 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 603/1243] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:37:05 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:37:05.816064 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb86809f-997a-4f96-87ef-eca65bc685c1 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:05.912717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:05.941613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb86809f-997a-4f96-87ef-eca65bc685c1 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:05.982120 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b50425-ca0e-4b8a-9f98-9c7cba6e4521 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:05.982762 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b50425-ca0e-4b8a-9f98-9c7cba6e4521 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:05.983043 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b50425-ca0e-4b8a-9f98-9c7cba6e4521 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:05.983319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b50425-ca0e-4b8a-9f98-9c7cba6e4521 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:05.991098 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b50425-ca0e-4b8a-9f98-9c7cba6e4521 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:05.992599 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b50425-ca0e-4b8a-9f98-9c7cba6e4521 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:06.004682 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-73b50425-ca0e-4b8a-9f98-9c7cba6e4521 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:37:06.004682 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:37:06.004682 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:37:06.004682 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:37:06.004682 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:37:06.004682 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:37:06.004682 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:37:06.004682 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:37:06.004682 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:37:06.004682 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:37:06.004682 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:37:06.004682 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:37:06.004682 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:37:06.004682 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:37:06.004682 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:37:06.005797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:37:06.005797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:37:06.005797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:37:06.005797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:37:06.005797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:37:06.005797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:37:06.005797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:37:06.005797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:37:06.005797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:37:06.005797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:37:06.005797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:37:06.005797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:37:06.005797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:37:06.005797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:37:06.005797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:37:06.005797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:37:06.005797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:37:06.006679 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:37:06.006679 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:37:06.006679 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:37:06.006679 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:37:06.006679 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:37:06.006679 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:37:06.006679 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:37:06.006679 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:37:06.006679 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 641/1244] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:37:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:37:06.012260 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954e3e43-d1e0-4728-9379-7cfa0df5769b None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:06.157946 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954e3e43-d1e0-4728-9379-7cfa0df5769b None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:06.201479 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f79e8496-4368-4406-926e-95a9d6125d9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:06.202157 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79e8496-4368-4406-926e-95a9d6125d9f None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:06.202429 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79e8496-4368-4406-926e-95a9d6125d9f None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:06.202709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79e8496-4368-4406-926e-95a9d6125d9f None admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:06.203505 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f79e8496-4368-4406-926e-95a9d6125d9f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:37:06.203840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f79e8496-4368-4406-926e-95a9d6125d9f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:06.205498 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f79e8496-4368-4406-926e-95a9d6125d9f None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:06.223848 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 604/1245] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:37:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:37:06.226092 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b50425-ca0e-4b8a-9f98-9c7cba6e4521 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:06.350031 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b50425-ca0e-4b8a-9f98-9c7cba6e4521 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:06.389076 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a94176-c9b0-4f64-81b4-b6ddbcdbd7d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:06.389697 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a94176-c9b0-4f64-81b4-b6ddbcdbd7d6 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:06.390171 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a94176-c9b0-4f64-81b4-b6ddbcdbd7d6 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:06.390624 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a94176-c9b0-4f64-81b4-b6ddbcdbd7d6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:06.391373 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a94176-c9b0-4f64-81b4-b6ddbcdbd7d6 None admin] RBAC: Adding query filter params (role.id=628bab09d1dc48fd9e01023b2f7671f6, scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5, user.id=0ee0002a76c3471f819b37fef510ef99) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:37:06.391746 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a94176-c9b0-4f64-81b4-b6ddbcdbd7d6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:06.393425 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a94176-c9b0-4f64-81b4-b6ddbcdbd7d6 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:06.400757 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 642/1246] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:37:06 2022] GET /identity/v3/role_assignments?user.id=0ee0002a76c3471f819b37fef510ef99&scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5&role.id=628bab09d1dc48fd9e01023b2f7671f6 => generated 610 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:37:08.620207 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992b64b0-8a7a-4ada-b18b-ea686811ba57 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:08.620786 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992b64b0-8a7a-4ada-b18b-ea686811ba57 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:08.620786 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992b64b0-8a7a-4ada-b18b-ea686811ba57 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:08.622329 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 605/1247] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:37:08 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:37:08.629085 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169771d2-1309-447e-986c-c471721aeb86 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:08.629582 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169771d2-1309-447e-986c-c471721aeb86 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:08.629684 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169771d2-1309-447e-986c-c471721aeb86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:08.636134 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:08.653151 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-169771d2-1309-447e-986c-c471721aeb86 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:37:08.654728 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-169771d2-1309-447e-986c-c471721aeb86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:37:08.733320 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 643/1248] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:37:08 2022] POST /identity/v3/auth/tokens => generated 2123 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:37:09.051975 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992b64b0-8a7a-4ada-b18b-ea686811ba57 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:09.074758 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:09.185993 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992b64b0-8a7a-4ada-b18b-ea686811ba57 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:09.228153 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1911db2-dd11-4637-b6be-523d5f113b1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:09.228948 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1911db2-dd11-4637-b6be-523d5f113b1f None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:09.229548 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1911db2-dd11-4637-b6be-523d5f113b1f None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:09.229959 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1911db2-dd11-4637-b6be-523d5f113b1f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:09.243864 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1911db2-dd11-4637-b6be-523d5f113b1f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:37:09.250057 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-e1911db2-dd11-4637-b6be-523d5f113b1f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:37:09.250057 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:37:09.250057 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:37:09.250057 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:37:09.250057 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:37:09.250057 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:37:09.250057 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:37:09.250057 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:37:09.250057 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:37:09.250057 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:37:09.250057 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:37:09.250057 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:37:09.250057 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:37:09.250057 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:37:09.250057 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:37:09.250057 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:37:09.251378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:37:09.251378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:37:09.251378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:37:09.251378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:37:09.251378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:37:09.251378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:37:09.251378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:37:09.251378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:37:09.251378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:37:09.251378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:37:09.251378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:37:09.251378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:37:09.251378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:37:09.251378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:37:09.251378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:37:09.251378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:37:09.252476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:37:09.252476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:37:09.252476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:37:09.252476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:37:09.252476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:37:09.252476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:37:09.252476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:37:09.252476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:37:09.252476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:37:09.252476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:37:09.252476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:37:09.252476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:37:09.252476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 606/1249] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:37:09 2022] GET /identity/v3/roles/admin => generated 83 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:37:09.257546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-169771d2-1309-447e-986c-c471721aeb86 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:09.377450 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-169771d2-1309-447e-986c-c471721aeb86 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:09.413633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:09.421869 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6dea77-45a2-4d36-8e66-ee2b4c04829a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:09.422601 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6dea77-45a2-4d36-8e66-ee2b4c04829a None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:09.422890 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6dea77-45a2-4d36-8e66-ee2b4c04829a None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:09.423174 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6dea77-45a2-4d36-8e66-ee2b4c04829a None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:09.423977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6dea77-45a2-4d36-8e66-ee2b4c04829a None admin] RBAC: Adding query filter params (name=admin) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:37:09.424288 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6dea77-45a2-4d36-8e66-ee2b4c04829a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:09.425807 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6dea77-45a2-4d36-8e66-ee2b4c04829a None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:09.435583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 644/1250] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:37:09 2022] GET /identity/v3/roles?name=admin => generated 341 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:37:09.442291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1911db2-dd11-4637-b6be-523d5f113b1f None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:09.530850 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:09.557854 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1911db2-dd11-4637-b6be-523d5f113b1f None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:09.598587 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736e7a05-8fd7-4dba-9975-247545407b5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:09.599554 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736e7a05-8fd7-4dba-9975-247545407b5d None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:09.599962 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736e7a05-8fd7-4dba-9975-247545407b5d None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:09.600478 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736e7a05-8fd7-4dba-9975-247545407b5d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:09.614445 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736e7a05-8fd7-4dba-9975-247545407b5d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:09.616787 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736e7a05-8fd7-4dba-9975-247545407b5d None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:09.633629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-736e7a05-8fd7-4dba-9975-247545407b5d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:37:09.633629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:37:09.633629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:37:09.633629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:37:09.633629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:37:09.633629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:37:09.633629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:37:09.633629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:37:09.633629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:37:09.633629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:37:09.633629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:37:09.633629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:37:09.633629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:37:09.633629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:37:09.633629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:37:09.633629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:37:09.635483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:37:09.635483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:37:09.635483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:37:09.635483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:37:09.635483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:37:09.635483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:37:09.635483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:37:09.635483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:37:09.635483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:37:09.635483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:37:09.635483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:37:09.635483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:37:09.635483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:37:09.635483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:37:09.635483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:37:09.635483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:37:09.635483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:37:09.636882 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:37:09.636882 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:37:09.636882 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:37:09.636882 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 607/1251] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:37:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:37:09.641122 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6dea77-45a2-4d36-8e66-ee2b4c04829a None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:09.761810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6dea77-45a2-4d36-8e66-ee2b4c04829a None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:09.801191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01835561-27c5-4b19-b16d-97ee2ca67d3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:09.801941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01835561-27c5-4b19-b16d-97ee2ca67d3f None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:09.802217 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01835561-27c5-4b19-b16d-97ee2ca67d3f None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:09.802541 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01835561-27c5-4b19-b16d-97ee2ca67d3f None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:09.803194 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01835561-27c5-4b19-b16d-97ee2ca67d3f None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:37:09.803770 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01835561-27c5-4b19-b16d-97ee2ca67d3f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:09.805641 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01835561-27c5-4b19-b16d-97ee2ca67d3f None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:09.818516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 645/1252] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:37:09 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:37:09.825530 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736e7a05-8fd7-4dba-9975-247545407b5d None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:09.947224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736e7a05-8fd7-4dba-9975-247545407b5d None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:09.976001 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:09.984997 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f21942-752e-4187-a438-53d1df163620 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:09.985790 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f21942-752e-4187-a438-53d1df163620 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:09.985940 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f21942-752e-4187-a438-53d1df163620 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:09.986309 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f21942-752e-4187-a438-53d1df163620 None admin] PATH_INFO: `/v3/users/placement` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:09.995131 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f21942-752e-4187-a438-53d1df163620 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:09.996685 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f21942-752e-4187-a438-53d1df163620 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:10.008899 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-e6f21942-752e-4187-a438-53d1df163620 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 24 15:37:10.008899 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:37:10.008899 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:37:10.008899 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:37:10.008899 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:37:10.008899 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:37:10.008899 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:37:10.008899 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:37:10.008899 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:37:10.008899 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:37:10.008899 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:37:10.008899 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:37:10.008899 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:37:10.008899 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:37:10.008899 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:37:10.008899 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:37:10.010106 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:37:10.010106 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:37:10.010106 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:37:10.010106 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:37:10.010106 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:37:10.010106 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:37:10.010106 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:37:10.010106 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:37:10.010106 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:37:10.010106 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:37:10.010106 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:37:10.010106 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:37:10.010106 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:37:10.010106 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:37:10.010106 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:37:10.010106 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:37:10.010106 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:37:10.010106 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:37:10.011354 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:37:10.011354 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:37:10.011354 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:37:10.011354 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:37:10.011354 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:37:10.011354 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:37:10.011354 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:37:10.011354 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:37:10.011354 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 24 15:37:10.011354 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:37:10.011354 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 608/1253] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:37:09 2022] GET /identity/v3/users/placement => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:37:10.017192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01835561-27c5-4b19-b16d-97ee2ca67d3f None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:10.090517 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:10.139343 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01835561-27c5-4b19-b16d-97ee2ca67d3f None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:10.179781 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42acff78-b84e-4c28-99ff-8d8a11aad5e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:10.180769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42acff78-b84e-4c28-99ff-8d8a11aad5e1 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:10.181261 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42acff78-b84e-4c28-99ff-8d8a11aad5e1 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:10.181787 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42acff78-b84e-4c28-99ff-8d8a11aad5e1 None admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:10.183187 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42acff78-b84e-4c28-99ff-8d8a11aad5e1 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:37:10.183771 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42acff78-b84e-4c28-99ff-8d8a11aad5e1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:10.186977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42acff78-b84e-4c28-99ff-8d8a11aad5e1 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:10.208899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 646/1254] 10.222.0.102 () {56 vars in 1148 bytes} [Wed Aug 24 15:37:10 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 380 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:37:10.216076 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f21942-752e-4187-a438-53d1df163620 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:10.343918 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f21942-752e-4187-a438-53d1df163620 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:10.382270 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34898fed-265b-4371-9f41-c065d22eb4fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:10.382819 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34898fed-265b-4371-9f41-c065d22eb4fc None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:10.383080 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34898fed-265b-4371-9f41-c065d22eb4fc None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:10.383346 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34898fed-265b-4371-9f41-c065d22eb4fc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:10.451594 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:10.469456 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34898fed-265b-4371-9f41-c065d22eb4fc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:10.470940 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34898fed-265b-4371-9f41-c065d22eb4fc None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:10.588631 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 609/1255] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:37:10 2022] GET /identity/v3/auth/tokens => generated 2123 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 15:37:10.594819 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42acff78-b84e-4c28-99ff-8d8a11aad5e1 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:10.710464 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42acff78-b84e-4c28-99ff-8d8a11aad5e1 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:10.727500 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:10.757480 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e15ce5-9cc6-4886-8e06-6f2480bd4219 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:10.758150 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e15ce5-9cc6-4886-8e06-6f2480bd4219 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:10.758408 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e15ce5-9cc6-4886-8e06-6f2480bd4219 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:10.758687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e15ce5-9cc6-4886-8e06-6f2480bd4219 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:10.771605 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e15ce5-9cc6-4886-8e06-6f2480bd4219 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:10.773102 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e15ce5-9cc6-4886-8e06-6f2480bd4219 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:10.789633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-01e15ce5-9cc6-4886-8e06-6f2480bd4219 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:37:10.789633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:37:10.789633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:37:10.789633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:37:10.789633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:37:10.789633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:37:10.789633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:37:10.789633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:37:10.789633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:37:10.789633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:37:10.789633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:37:10.789633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:37:10.789633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:37:10.789633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:37:10.789633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:37:10.789633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:37:10.791178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:37:10.791178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:37:10.791178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:37:10.791178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:37:10.791178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:37:10.791178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:37:10.791178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:37:10.791178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:37:10.791178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:37:10.791178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:37:10.791178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:37:10.791178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:37:10.791178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:37:10.791178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:37:10.791178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:37:10.791178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:37:10.791178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:37:10.792120 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:37:10.792120 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:37:10.792120 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:37:10.792120 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 647/1256] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:37:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:37:10.797855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34898fed-265b-4371-9f41-c065d22eb4fc None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:10.901789 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:10.919160 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34898fed-265b-4371-9f41-c065d22eb4fc None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:10.958982 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697754d9-d5ea-4ec9-86b9-7bd6cd1b9230 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:10.959591 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697754d9-d5ea-4ec9-86b9-7bd6cd1b9230 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:10.959925 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697754d9-d5ea-4ec9-86b9-7bd6cd1b9230 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:10.960185 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697754d9-d5ea-4ec9-86b9-7bd6cd1b9230 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:10.960831 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-697754d9-d5ea-4ec9-86b9-7bd6cd1b9230 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:37:10.961134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-697754d9-d5ea-4ec9-86b9-7bd6cd1b9230 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:10.962505 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-697754d9-d5ea-4ec9-86b9-7bd6cd1b9230 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:10.975523 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 610/1257] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:37:10 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:37:10.981342 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e15ce5-9cc6-4886-8e06-6f2480bd4219 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:11.116070 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e15ce5-9cc6-4886-8e06-6f2480bd4219 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:11.163980 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5917492-1703-4407-8bd5-20b30975d010 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:11.165264 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5917492-1703-4407-8bd5-20b30975d010 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:11.165674 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5917492-1703-4407-8bd5-20b30975d010 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:11.166052 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5917492-1703-4407-8bd5-20b30975d010 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:11.174099 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5917492-1703-4407-8bd5-20b30975d010 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:11.175890 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5917492-1703-4407-8bd5-20b30975d010 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:11.191435 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-f5917492-1703-4407-8bd5-20b30975d010 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:37:11.191435 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:37:11.191435 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:37:11.191435 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:37:11.191435 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:37:11.191435 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:37:11.191435 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:37:11.191435 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:37:11.191435 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:37:11.191435 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:37:11.191435 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:37:11.191435 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:37:11.191435 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:37:11.191435 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:37:11.191435 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:37:11.192720 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:37:11.192720 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:37:11.192720 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:37:11.192720 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:37:11.192720 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:37:11.192720 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:37:11.192720 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:37:11.192720 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:37:11.192720 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:37:11.192720 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:37:11.192720 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:37:11.192720 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:37:11.192720 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:37:11.192720 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:37:11.192720 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:37:11.192720 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:37:11.192720 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:37:11.193947 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:37:11.193947 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:37:11.193947 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:37:11.193947 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:37:11.193947 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:37:11.193947 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:37:11.193947 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:37:11.193947 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:37:11.193947 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 648/1258] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:37:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:37:11.198896 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697754d9-d5ea-4ec9-86b9-7bd6cd1b9230 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:11.322074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697754d9-d5ea-4ec9-86b9-7bd6cd1b9230 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:11.365069 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4015b0c5-a49a-4c49-a0b5-cec6551f2e5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:11.365682 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4015b0c5-a49a-4c49-a0b5-cec6551f2e5d None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:11.365963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4015b0c5-a49a-4c49-a0b5-cec6551f2e5d None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:11.366235 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4015b0c5-a49a-4c49-a0b5-cec6551f2e5d None admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:11.366934 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4015b0c5-a49a-4c49-a0b5-cec6551f2e5d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:37:11.367247 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4015b0c5-a49a-4c49-a0b5-cec6551f2e5d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:11.368739 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4015b0c5-a49a-4c49-a0b5-cec6551f2e5d None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:11.383306 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 611/1259] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:37:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:37:11.389576 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5917492-1703-4407-8bd5-20b30975d010 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:11.425220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:11.510754 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5917492-1703-4407-8bd5-20b30975d010 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:11.549620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df1882b-b1a4-4a9b-a834-fa5ff3704a84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:11.550673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df1882b-b1a4-4a9b-a834-fa5ff3704a84 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:11.551032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df1882b-b1a4-4a9b-a834-fa5ff3704a84 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:11.551591 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df1882b-b1a4-4a9b-a834-fa5ff3704a84 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:11.552507 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df1882b-b1a4-4a9b-a834-fa5ff3704a84 None admin] RBAC: Adding query filter params (role.id=4d8fbbaea81a4145b89af12350bb067c, scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5, user.id=0ee0002a76c3471f819b37fef510ef99) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:37:11.552839 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df1882b-b1a4-4a9b-a834-fa5ff3704a84 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:11.555436 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df1882b-b1a4-4a9b-a834-fa5ff3704a84 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:11.575745 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 649/1260] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:37:11 2022] GET /identity/v3/role_assignments?user.id=0ee0002a76c3471f819b37fef510ef99&scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5&role.id=4d8fbbaea81a4145b89af12350bb067c => generated 260 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:37:13.487079 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068cc200-b3d1-482a-99be-cdedcb40b534 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:13.487660 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068cc200-b3d1-482a-99be-cdedcb40b534 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:13.487660 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068cc200-b3d1-482a-99be-cdedcb40b534 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:13.488995 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 612/1261] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:37:13 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:37:13.494153 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf43421-c7d9-416f-a21c-9e739ba072b3 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:13.494707 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf43421-c7d9-416f-a21c-9e739ba072b3 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:13.494707 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf43421-c7d9-416f-a21c-9e739ba072b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:13.512610 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-9cf43421-c7d9-416f-a21c-9e739ba072b3 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:37:13.513798 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-9cf43421-c7d9-416f-a21c-9e739ba072b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:37:13.589812 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 650/1262] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:37:13 2022] POST /identity/v3/auth/tokens => generated 2123 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:37:13.862808 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068cc200-b3d1-482a-99be-cdedcb40b534 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:13.874588 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:13.969627 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068cc200-b3d1-482a-99be-cdedcb40b534 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:14.006422 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1ea6af0-ae00-49a6-8ef9-524c0c48cf42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:14.007082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ea6af0-ae00-49a6-8ef9-524c0c48cf42 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:14.007378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ea6af0-ae00-49a6-8ef9-524c0c48cf42 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:14.007985 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ea6af0-ae00-49a6-8ef9-524c0c48cf42 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:14.025510 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1ea6af0-ae00-49a6-8ef9-524c0c48cf42 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:37:14.033480 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-d1ea6af0-ae00-49a6-8ef9-524c0c48cf42 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:37:14.033480 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:37:14.033480 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:37:14.033480 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:37:14.033480 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:37:14.033480 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:37:14.033480 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:37:14.033480 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:37:14.033480 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:37:14.033480 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:37:14.033480 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:37:14.033480 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:37:14.033480 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:37:14.033480 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:37:14.033480 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:37:14.033480 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:37:14.034562 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:37:14.034562 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:37:14.034562 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:37:14.034562 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:37:14.034562 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:37:14.034562 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:37:14.034562 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:37:14.034562 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:37:14.034562 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:37:14.034562 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:37:14.034562 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:37:14.034562 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:37:14.034562 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:37:14.034562 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:37:14.034562 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:37:14.034562 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:37:14.035414 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:37:14.035414 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:37:14.035414 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:37:14.035414 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:37:14.035414 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:37:14.035414 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:37:14.035414 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:37:14.035414 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:37:14.035414 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:37:14.035414 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:37:14.035414 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:37:14.035414 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:37:14.035414 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 613/1263] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:37:13 2022] GET /identity/v3/roles/admin => generated 83 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:37:14.042024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf43421-c7d9-416f-a21c-9e739ba072b3 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:14.067089 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:14.152674 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf43421-c7d9-416f-a21c-9e739ba072b3 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:14.187996 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5192ac-b286-4ec8-8600-338e0ab1ed60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:14.188761 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5192ac-b286-4ec8-8600-338e0ab1ed60 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:14.189108 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5192ac-b286-4ec8-8600-338e0ab1ed60 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:14.189387 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5192ac-b286-4ec8-8600-338e0ab1ed60 None admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:14.190189 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad5192ac-b286-4ec8-8600-338e0ab1ed60 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:37:14.190494 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad5192ac-b286-4ec8-8600-338e0ab1ed60 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:14.191896 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad5192ac-b286-4ec8-8600-338e0ab1ed60 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:14.200183 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 651/1264] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:37:14 2022] GET /identity/v3/roles?name=admin => generated 341 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:37:14.206156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1ea6af0-ae00-49a6-8ef9-524c0c48cf42 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:14.278265 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:14.312603 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1ea6af0-ae00-49a6-8ef9-524c0c48cf42 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:14.346066 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dfa1c57-7dad-4799-a832-f2854e5deb5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:14.346690 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfa1c57-7dad-4799-a832-f2854e5deb5b None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:14.346970 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfa1c57-7dad-4799-a832-f2854e5deb5b None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:14.347241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfa1c57-7dad-4799-a832-f2854e5deb5b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:14.357070 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dfa1c57-7dad-4799-a832-f2854e5deb5b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:14.359237 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dfa1c57-7dad-4799-a832-f2854e5deb5b None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:14.374134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-8dfa1c57-7dad-4799-a832-f2854e5deb5b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:37:14.374134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:37:14.374134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:37:14.374134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:37:14.374134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:37:14.374134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:37:14.374134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:37:14.374134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:37:14.374134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:37:14.374134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:37:14.374134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:37:14.374134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:37:14.374134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:37:14.374134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:37:14.374134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:37:14.374134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:37:14.375259 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:37:14.375259 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:37:14.375259 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:37:14.375259 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:37:14.375259 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:37:14.375259 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:37:14.375259 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:37:14.375259 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:37:14.375259 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:37:14.375259 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:37:14.375259 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:37:14.375259 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:37:14.375259 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:37:14.375259 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:37:14.375259 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:37:14.375259 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:37:14.375259 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:37:14.376331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:37:14.376331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:37:14.376331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:37:14.376331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 614/1265] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:37:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:37:14.381265 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5192ac-b286-4ec8-8600-338e0ab1ed60 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:14.461910 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:14.480533 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5192ac-b286-4ec8-8600-338e0ab1ed60 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:14.514084 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1aab48-b416-4243-ba68-b15354b490c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:14.514703 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1aab48-b416-4243-ba68-b15354b490c9 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:14.514939 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1aab48-b416-4243-ba68-b15354b490c9 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:14.515266 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1aab48-b416-4243-ba68-b15354b490c9 None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:14.515866 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1aab48-b416-4243-ba68-b15354b490c9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:37:14.516167 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1aab48-b416-4243-ba68-b15354b490c9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:14.517570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1aab48-b416-4243-ba68-b15354b490c9 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:14.529037 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 652/1266] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:37:14 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:37:14.534776 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dfa1c57-7dad-4799-a832-f2854e5deb5b None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:14.634492 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dfa1c57-7dad-4799-a832-f2854e5deb5b None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:14.654134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:14.668849 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80df92bb-818f-4e26-a77d-22e90edb3ece None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:14.669469 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80df92bb-818f-4e26-a77d-22e90edb3ece None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:14.669746 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80df92bb-818f-4e26-a77d-22e90edb3ece None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:14.670016 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80df92bb-818f-4e26-a77d-22e90edb3ece None admin] PATH_INFO: `/v3/users/placement` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:14.678536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80df92bb-818f-4e26-a77d-22e90edb3ece None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:14.680000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80df92bb-818f-4e26-a77d-22e90edb3ece None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:14.690954 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-80df92bb-818f-4e26-a77d-22e90edb3ece None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 24 15:37:14.690954 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:37:14.690954 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:37:14.690954 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:37:14.690954 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:37:14.690954 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:37:14.690954 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:37:14.690954 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:37:14.690954 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:37:14.690954 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:37:14.690954 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:37:14.690954 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:37:14.690954 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:37:14.690954 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:37:14.690954 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:37:14.690954 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:37:14.692142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:37:14.692142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:37:14.692142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:37:14.692142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:37:14.692142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:37:14.692142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:37:14.692142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:37:14.692142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:37:14.692142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:37:14.692142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:37:14.692142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:37:14.692142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:37:14.692142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:37:14.692142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:37:14.692142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:37:14.692142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:37:14.692142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:37:14.692142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:37:14.693276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:37:14.693276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:37:14.693276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:37:14.693276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:37:14.693276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:37:14.693276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:37:14.693276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:37:14.693276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:37:14.693276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 24 15:37:14.693276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:37:14.693276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 615/1267] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:37:14 2022] GET /identity/v3/users/placement => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 15:37:14.697859 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1aab48-b416-4243-ba68-b15354b490c9 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:14.798962 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1aab48-b416-4243-ba68-b15354b490c9 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:14.834343 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70b823e-c7a9-497a-a4d9-26ee828614c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:14.834962 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70b823e-c7a9-497a-a4d9-26ee828614c5 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:14.835242 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70b823e-c7a9-497a-a4d9-26ee828614c5 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:14.835516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70b823e-c7a9-497a-a4d9-26ee828614c5 None admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:14.836280 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a70b823e-c7a9-497a-a4d9-26ee828614c5 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:37:14.836588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a70b823e-c7a9-497a-a4d9-26ee828614c5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:14.838003 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a70b823e-c7a9-497a-a4d9-26ee828614c5 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:14.847175 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:14.856573 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 653/1268] 10.222.0.102 () {56 vars in 1148 bytes} [Wed Aug 24 15:37:14 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 380 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:37:14.862528 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80df92bb-818f-4e26-a77d-22e90edb3ece None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:14.964652 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80df92bb-818f-4e26-a77d-22e90edb3ece None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:15.000398 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c2b06a-046a-4f08-8a1b-01cc50c994b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:15.001025 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c2b06a-046a-4f08-8a1b-01cc50c994b3 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:15.001304 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c2b06a-046a-4f08-8a1b-01cc50c994b3 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:15.001572 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c2b06a-046a-4f08-8a1b-01cc50c994b3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:15.013592 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c2b06a-046a-4f08-8a1b-01cc50c994b3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:15.015710 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c2b06a-046a-4f08-8a1b-01cc50c994b3 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:15.033082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-00c2b06a-046a-4f08-8a1b-01cc50c994b3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:37:15.033082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:37:15.033082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:37:15.033082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:37:15.033082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:37:15.033082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:37:15.033082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:37:15.033082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:37:15.033082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:37:15.033082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:37:15.033082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:37:15.033082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:37:15.033082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:37:15.033082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:37:15.033082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:37:15.033082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:37:15.034564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:37:15.034564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:37:15.034564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:37:15.034564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:37:15.034564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:37:15.034564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:37:15.034564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:37:15.034564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:37:15.034564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:37:15.034564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:37:15.034564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:37:15.034564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:37:15.034564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:37:15.034564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:37:15.034564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:37:15.034564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:37:15.034564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:37:15.035557 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:37:15.035557 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:37:15.035557 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:37:15.035557 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 616/1269] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:37:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:37:15.040609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70b823e-c7a9-497a-a4d9-26ee828614c5 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:15.142749 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70b823e-c7a9-497a-a4d9-26ee828614c5 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:15.176448 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c9d23b-1447-417d-bba4-48c5cf91c23c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:15.177048 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c9d23b-1447-417d-bba4-48c5cf91c23c None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:15.177331 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c9d23b-1447-417d-bba4-48c5cf91c23c None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:15.177603 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c9d23b-1447-417d-bba4-48c5cf91c23c None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:15.178511 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c9d23b-1447-417d-bba4-48c5cf91c23c None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:37:15.178814 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c9d23b-1447-417d-bba4-48c5cf91c23c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:15.180223 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c9d23b-1447-417d-bba4-48c5cf91c23c None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:15.192510 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 654/1270] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:37:15 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:37:15.198222 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c2b06a-046a-4f08-8a1b-01cc50c994b3 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:15.230089 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:15.304053 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c2b06a-046a-4f08-8a1b-01cc50c994b3 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:15.337396 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5afa1f-364c-432a-baf3-d053721e81e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:15.338020 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5afa1f-364c-432a-baf3-d053721e81e3 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:15.338298 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5afa1f-364c-432a-baf3-d053721e81e3 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:15.338575 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5afa1f-364c-432a-baf3-d053721e81e3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:15.345872 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb5afa1f-364c-432a-baf3-d053721e81e3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:15.347331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb5afa1f-364c-432a-baf3-d053721e81e3 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:15.360280 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-bb5afa1f-364c-432a-baf3-d053721e81e3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:37:15.360280 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:37:15.360280 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:37:15.360280 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:37:15.360280 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:37:15.360280 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:37:15.360280 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:37:15.360280 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:37:15.360280 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:37:15.360280 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:37:15.360280 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:37:15.360280 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:37:15.360280 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:37:15.360280 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:37:15.360280 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:37:15.361331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:37:15.361331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:37:15.361331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:37:15.361331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:37:15.361331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:37:15.361331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:37:15.361331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:37:15.361331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:37:15.361331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:37:15.361331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:37:15.361331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:37:15.361331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:37:15.361331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:37:15.361331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:37:15.361331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:37:15.361331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:37:15.361331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:37:15.362571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:37:15.362571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:37:15.362571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:37:15.362571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:37:15.362571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:37:15.362571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:37:15.362571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:37:15.362571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:37:15.362571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 617/1271] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:37:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:37:15.367835 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c9d23b-1447-417d-bba4-48c5cf91c23c None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:15.426856 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:15.468941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c9d23b-1447-417d-bba4-48c5cf91c23c None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:15.502115 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a32be9-317e-419b-8b06-2893275261c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:15.502714 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a32be9-317e-419b-8b06-2893275261c5 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:15.502998 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a32be9-317e-419b-8b06-2893275261c5 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:15.503298 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a32be9-317e-419b-8b06-2893275261c5 None admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:15.504017 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a32be9-317e-419b-8b06-2893275261c5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:37:15.504330 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a32be9-317e-419b-8b06-2893275261c5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:15.505755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a32be9-317e-419b-8b06-2893275261c5 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:15.517801 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 655/1272] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:37:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:37:15.523864 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5afa1f-364c-432a-baf3-d053721e81e3 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:15.615045 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:15.624004 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5afa1f-364c-432a-baf3-d053721e81e3 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:15.658950 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edcda8d5-7ffb-46d0-8b01-92d44094f32f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:15.659600 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcda8d5-7ffb-46d0-8b01-92d44094f32f None admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:15.659911 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcda8d5-7ffb-46d0-8b01-92d44094f32f None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:15.660188 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcda8d5-7ffb-46d0-8b01-92d44094f32f None admin] PATH_INFO: `/v3/projects/3e50e6f6c1f44c648337c8d37a8e94a5/users/0ee0002a76c3471f819b37fef510ef99/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:15.662060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edcda8d5-7ffb-46d0-8b01-92d44094f32f None admin] RBAC: Authorizing `identity:create_grant(project_id=3e50e6f6c1f44c648337c8d37a8e94a5, user_id=0ee0002a76c3471f819b37fef510ef99, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:15.663489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edcda8d5-7ffb-46d0-8b01-92d44094f32f None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:15.677067 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 618/1273] 10.222.0.102 () {58 vars in 1444 bytes} [Wed Aug 24 15:37:15 2022] PUT /identity/v3/projects/3e50e6f6c1f44c648337c8d37a8e94a5/users/0ee0002a76c3471f819b37fef510ef99/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:37:17.584358 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4df1d7-3b1b-448d-a0f1-081762354719 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:17.584890 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4df1d7-3b1b-448d-a0f1-081762354719 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:17.584890 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4df1d7-3b1b-448d-a0f1-081762354719 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:17.586257 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 656/1274] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:37:17 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:37:17.591697 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4368d40f-5a03-4b24-b087-88bdeb2cd0c0 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:17.592128 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4368d40f-5a03-4b24-b087-88bdeb2cd0c0 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:17.592252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4368d40f-5a03-4b24-b087-88bdeb2cd0c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:17.611279 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-4368d40f-5a03-4b24-b087-88bdeb2cd0c0 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:37:17.612492 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-4368d40f-5a03-4b24-b087-88bdeb2cd0c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:37:17.678592 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 619/1275] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:37:17 2022] POST /identity/v3/auth/tokens => generated 2123 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:37:17.948921 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d4df1d7-3b1b-448d-a0f1-081762354719 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:18.071101 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d4df1d7-3b1b-448d-a0f1-081762354719 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:18.091797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:18.109312 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a16e99b-acee-412e-9e91-e441be437a86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:18.109860 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a16e99b-acee-412e-9e91-e441be437a86 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:18.110138 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a16e99b-acee-412e-9e91-e441be437a86 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:18.110404 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a16e99b-acee-412e-9e91-e441be437a86 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:18.120270 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a16e99b-acee-412e-9e91-e441be437a86 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 15:37:18.124978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-8a16e99b-acee-412e-9e91-e441be437a86 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:37:18.124978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:37:18.124978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:37:18.124978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:37:18.124978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:37:18.124978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:37:18.124978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:37:18.124978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:37:18.124978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:37:18.124978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:37:18.124978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:37:18.124978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:37:18.124978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 15:37:18.124978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 15:37:18.124978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 15:37:18.124978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 15:37:18.126104 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 15:37:18.126104 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 15:37:18.126104 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 15:37:18.126104 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 15:37:18.126104 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:37:18.126104 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:37:18.126104 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:37:18.126104 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:37:18.126104 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:37:18.126104 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:37:18.126104 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:37:18.126104 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:37:18.126104 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:37:18.126104 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:37:18.126104 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:37:18.126104 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:37:18.127196 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:37:18.127196 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:37:18.127196 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:37:18.127196 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:37:18.127196 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 15:37:18.127196 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 15:37:18.127196 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 15:37:18.127196 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 15:37:18.127196 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 15:37:18.127196 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 15:37:18.127196 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 15:37:18.127196 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:37:18.131032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 657/1276] 10.222.0.102 () {56 vars in 1101 bytes} [Wed Aug 24 15:37:17 2022] GET /identity/v3/roles/admin => generated 83 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:37:18.132315 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4368d40f-5a03-4b24-b087-88bdeb2cd0c0 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:18.212401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:18.232169 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4368d40f-5a03-4b24-b087-88bdeb2cd0c0 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:18.265495 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8d3cd1-ed1a-4ea8-ab59-689351bfbce8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:18.266179 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8d3cd1-ed1a-4ea8-ab59-689351bfbce8 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:18.266452 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8d3cd1-ed1a-4ea8-ab59-689351bfbce8 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:18.266717 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8d3cd1-ed1a-4ea8-ab59-689351bfbce8 None admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:18.267382 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8d3cd1-ed1a-4ea8-ab59-689351bfbce8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:37:18.267779 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8d3cd1-ed1a-4ea8-ab59-689351bfbce8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:18.269133 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8d3cd1-ed1a-4ea8-ab59-689351bfbce8 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:18.277564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 620/1277] 10.222.0.102 () {56 vars in 1104 bytes} [Wed Aug 24 15:37:18 2022] GET /identity/v3/roles?name=admin => generated 341 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:37:18.284045 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a16e99b-acee-412e-9e91-e441be437a86 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:18.393844 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a16e99b-acee-412e-9e91-e441be437a86 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:18.429447 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9c1e2d-8d0e-44e7-8a81-7f5201276d39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:18.430065 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9c1e2d-8d0e-44e7-8a81-7f5201276d39 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:18.430339 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9c1e2d-8d0e-44e7-8a81-7f5201276d39 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:18.430604 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9c1e2d-8d0e-44e7-8a81-7f5201276d39 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:18.441727 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9c1e2d-8d0e-44e7-8a81-7f5201276d39 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:18.443184 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9c1e2d-8d0e-44e7-8a81-7f5201276d39 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:18.457493 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-ee9c1e2d-8d0e-44e7-8a81-7f5201276d39 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:37:18.457493 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:37:18.457493 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:37:18.457493 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:37:18.457493 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:37:18.457493 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:37:18.457493 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:37:18.457493 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:37:18.457493 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:37:18.457493 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:37:18.457493 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:37:18.457493 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:37:18.457493 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:37:18.457493 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:37:18.457493 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:37:18.457493 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:37:18.458725 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:37:18.458725 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:37:18.458725 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:37:18.458725 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:37:18.458725 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:37:18.458725 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:37:18.458725 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:37:18.458725 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:37:18.458725 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:37:18.458725 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:37:18.458725 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:37:18.458725 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:37:18.458725 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:37:18.458725 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:37:18.458725 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:37:18.458725 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:37:18.458725 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:37:18.459918 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:37:18.459918 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:37:18.459918 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:37:18.459918 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 658/1278] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:37:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:37:18.464544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8d3cd1-ed1a-4ea8-ab59-689351bfbce8 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:18.565300 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8d3cd1-ed1a-4ea8-ab59-689351bfbce8 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:18.599167 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a3d9de-24f5-4834-b969-5b33426148b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:18.599846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a3d9de-24f5-4834-b969-5b33426148b6 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:18.600076 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a3d9de-24f5-4834-b969-5b33426148b6 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:18.600350 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a3d9de-24f5-4834-b969-5b33426148b6 None admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:18.601000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a3d9de-24f5-4834-b969-5b33426148b6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:37:18.601305 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a3d9de-24f5-4834-b969-5b33426148b6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:18.602659 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a3d9de-24f5-4834-b969-5b33426148b6 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:18.614571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 621/1279] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:37:18 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:37:18.620311 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9c1e2d-8d0e-44e7-8a81-7f5201276d39 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:18.652890 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:18.725515 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9c1e2d-8d0e-44e7-8a81-7f5201276d39 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:18.760854 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750b94c0-3978-4a73-b264-8db0ba33e4f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:18.761482 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750b94c0-3978-4a73-b264-8db0ba33e4f8 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:18.761764 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750b94c0-3978-4a73-b264-8db0ba33e4f8 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:18.762040 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750b94c0-3978-4a73-b264-8db0ba33e4f8 None admin] PATH_INFO: `/v3/users/placement` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:18.769834 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-750b94c0-3978-4a73-b264-8db0ba33e4f8 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:18.771278 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-750b94c0-3978-4a73-b264-8db0ba33e4f8 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:18.782304 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-750b94c0-3978-4a73-b264-8db0ba33e4f8 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 24 15:37:18.782304 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:37:18.782304 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:37:18.782304 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:37:18.782304 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:37:18.782304 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:37:18.782304 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:37:18.782304 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:37:18.782304 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:37:18.782304 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:37:18.782304 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:37:18.782304 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:37:18.782304 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 15:37:18.782304 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 15:37:18.782304 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 15:37:18.782304 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 15:37:18.788063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:37:18.788063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:37:18.788063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 15:37:18.788063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:37:18.788063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 15:37:18.788063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 15:37:18.788063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:37:18.788063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:37:18.788063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:37:18.788063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:37:18.788063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:37:18.788063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:37:18.788063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:37:18.788063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:37:18.788063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:37:18.788063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:37:18.788063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:37:18.788063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:37:18.790042 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:37:18.790042 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:37:18.790042 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 15:37:18.790042 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 15:37:18.790042 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 15:37:18.790042 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 15:37:18.790042 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 15:37:18.790042 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 15:37:18.790042 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 24 15:37:18.790042 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:37:18.790042 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 659/1280] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:37:18 2022] GET /identity/v3/users/placement => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:37:18.790845 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a3d9de-24f5-4834-b969-5b33426148b6 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:18.795170 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:18.895657 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a3d9de-24f5-4834-b969-5b33426148b6 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:18.930631 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecc913f-47a5-43e3-a655-08b7d6923b8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:18.931142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecc913f-47a5-43e3-a655-08b7d6923b8c None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:18.931422 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecc913f-47a5-43e3-a655-08b7d6923b8c None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:18.931722 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecc913f-47a5-43e3-a655-08b7d6923b8c None admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:18.932455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecc913f-47a5-43e3-a655-08b7d6923b8c None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:37:18.932766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecc913f-47a5-43e3-a655-08b7d6923b8c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:18.934215 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecc913f-47a5-43e3-a655-08b7d6923b8c None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:18.952913 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 622/1281] 10.222.0.102 () {56 vars in 1148 bytes} [Wed Aug 24 15:37:18 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 380 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:37:18.959122 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750b94c0-3978-4a73-b264-8db0ba33e4f8 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:19.056866 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:19.066694 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750b94c0-3978-4a73-b264-8db0ba33e4f8 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:19.101394 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556d75c3-03af-4d3b-9c12-4754f32789aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:19.101992 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556d75c3-03af-4d3b-9c12-4754f32789aa None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:19.102275 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556d75c3-03af-4d3b-9c12-4754f32789aa None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:19.102550 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556d75c3-03af-4d3b-9c12-4754f32789aa None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:19.170503 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556d75c3-03af-4d3b-9c12-4754f32789aa None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:19.172721 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556d75c3-03af-4d3b-9c12-4754f32789aa None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:19.259802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:19.271607 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 660/1282] 10.222.0.102 () {58 vars in 1287 bytes} [Wed Aug 24 15:37:18 2022] GET /identity/v3/auth/tokens => generated 2123 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 15:37:19.277345 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecc913f-47a5-43e3-a655-08b7d6923b8c None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:19.341382 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:19.391477 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecc913f-47a5-43e3-a655-08b7d6923b8c None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:19.426289 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccdfdd23-a92f-47e6-a9cc-8cf4d5ef0261 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:19.426916 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdfdd23-a92f-47e6-a9cc-8cf4d5ef0261 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:19.427196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdfdd23-a92f-47e6-a9cc-8cf4d5ef0261 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:19.427465 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdfdd23-a92f-47e6-a9cc-8cf4d5ef0261 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:19.438400 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccdfdd23-a92f-47e6-a9cc-8cf4d5ef0261 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:19.439862 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccdfdd23-a92f-47e6-a9cc-8cf4d5ef0261 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:19.454566 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-ccdfdd23-a92f-47e6-a9cc-8cf4d5ef0261 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:37:19.454566 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:37:19.454566 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:37:19.454566 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:37:19.454566 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:37:19.454566 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:37:19.454566 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:37:19.454566 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:37:19.454566 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:37:19.454566 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:37:19.454566 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:37:19.454566 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:37:19.454566 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 15:37:19.454566 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 15:37:19.454566 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 15:37:19.454566 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 15:37:19.455771 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:37:19.455771 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:37:19.455771 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:37:19.455771 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:37:19.455771 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:37:19.455771 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:37:19.455771 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:37:19.455771 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:37:19.455771 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:37:19.455771 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:37:19.455771 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:37:19.455771 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:37:19.455771 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:37:19.455771 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:37:19.455771 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:37:19.455771 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:37:19.455771 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 15:37:19.456863 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 15:37:19.456863 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 15:37:19.456863 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:37:19.456863 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 623/1283] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:37:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:37:19.461338 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556d75c3-03af-4d3b-9c12-4754f32789aa None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:19.565513 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556d75c3-03af-4d3b-9c12-4754f32789aa None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:19.599449 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e5616d-f6a8-4f16-afcd-e1b5afed869e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:19.600080 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e5616d-f6a8-4f16-afcd-e1b5afed869e None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:19.600365 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e5616d-f6a8-4f16-afcd-e1b5afed869e None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:19.600641 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e5616d-f6a8-4f16-afcd-e1b5afed869e None admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:19.601285 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e5616d-f6a8-4f16-afcd-e1b5afed869e None admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:37:19.601591 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e5616d-f6a8-4f16-afcd-e1b5afed869e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:19.602964 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e5616d-f6a8-4f16-afcd-e1b5afed869e None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:19.615042 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 661/1284] 10.222.0.102 () {56 vars in 1114 bytes} [Wed Aug 24 15:37:19 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:37:19.620686 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccdfdd23-a92f-47e6-a9cc-8cf4d5ef0261 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:19.724986 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccdfdd23-a92f-47e6-a9cc-8cf4d5ef0261 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:19.744414 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:19.765326 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f473520c-2cc3-4b58-9dd6-1ec1a44d9b37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:19.765943 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f473520c-2cc3-4b58-9dd6-1ec1a44d9b37 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:19.766220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f473520c-2cc3-4b58-9dd6-1ec1a44d9b37 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:19.766487 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f473520c-2cc3-4b58-9dd6-1ec1a44d9b37 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:19.773338 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f473520c-2cc3-4b58-9dd6-1ec1a44d9b37 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:19.774759 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f473520c-2cc3-4b58-9dd6-1ec1a44d9b37 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:19.786403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application [None req-f473520c-2cc3-4b58-9dd6-1ec1a44d9b37 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:37:19.786403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:37:19.786403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:37:19.786403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:37:19.786403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:37:19.786403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:37:19.786403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:37:19.786403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:37:19.786403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:37:19.786403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:37:19.786403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:37:19.786403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:37:19.786403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 15:37:19.786403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 15:37:19.786403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 15:37:19.787683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 15:37:19.787683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:37:19.787683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:37:19.787683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:37:19.787683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:37:19.787683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:37:19.787683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:37:19.787683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:37:19.787683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application with Lock( Aug 24 15:37:19.787683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:37:19.787683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:37:19.787683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:37:19.787683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:37:19.787683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:37:19.787683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:37:19.787683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:37:19.787683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:37:19.788829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 15:37:19.788829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 15:37:19.788829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 15:37:19.788829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 15:37:19.788829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 15:37:19.788829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 15:37:19.788829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 15:37:19.788829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: ERROR keystone.server.flask.application Aug 24 15:37:19.788829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 624/1285] 10.222.0.102 () {56 vars in 1116 bytes} [Wed Aug 24 15:37:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:37:19.793424 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e5616d-f6a8-4f16-afcd-e1b5afed869e None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:19.845204 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:19.894575 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e5616d-f6a8-4f16-afcd-e1b5afed869e None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:19.929362 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24da7acc-c6cb-4d09-b89c-f926c0c7682c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:19.929965 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24da7acc-c6cb-4d09-b89c-f926c0c7682c None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:19.930241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24da7acc-c6cb-4d09-b89c-f926c0c7682c None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:19.930507 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24da7acc-c6cb-4d09-b89c-f926c0c7682c None admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:19.931186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24da7acc-c6cb-4d09-b89c-f926c0c7682c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:37:19.931528 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24da7acc-c6cb-4d09-b89c-f926c0c7682c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:19.932926 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24da7acc-c6cb-4d09-b89c-f926c0c7682c None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:19.944777 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 662/1286] 10.222.0.102 () {56 vars in 1153 bytes} [Wed Aug 24 15:37:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 431 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:37:19.951011 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f473520c-2cc3-4b58-9dd6-1ec1a44d9b37 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:20.073003 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f473520c-2cc3-4b58-9dd6-1ec1a44d9b37 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:20.106913 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53734f91-4ec3-44a1-9697-cd26f983e3f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:20.107427 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53734f91-4ec3-44a1-9697-cd26f983e3f1 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:20.107736 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53734f91-4ec3-44a1-9697-cd26f983e3f1 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:20.108009 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53734f91-4ec3-44a1-9697-cd26f983e3f1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:20.109622 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53734f91-4ec3-44a1-9697-cd26f983e3f1 None admin] RBAC: Adding query filter params (role.id=4d8fbbaea81a4145b89af12350bb067c, scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5, user.id=0ee0002a76c3471f819b37fef510ef99) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:37:20.109945 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53734f91-4ec3-44a1-9697-cd26f983e3f1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:20.111355 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53734f91-4ec3-44a1-9697-cd26f983e3f1 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:20.117061 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 625/1287] 10.222.0.102 () {56 vars in 1379 bytes} [Wed Aug 24 15:37:19 2022] GET /identity/v3/role_assignments?user.id=0ee0002a76c3471f819b37fef510ef99&scope.project.id=3e50e6f6c1f44c648337c8d37a8e94a5&role.id=4d8fbbaea81a4145b89af12350bb067c => generated 610 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:37:22.093068 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0f6aff-456b-4a0a-a198-8d1bd69c27f2 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:22.093733 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0f6aff-456b-4a0a-a198-8d1bd69c27f2 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:22.093733 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0f6aff-456b-4a0a-a198-8d1bd69c27f2 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:22.094956 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 663/1288] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:37:22 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:37:22.100498 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1f3f7d-b4de-4270-a411-fbf6b8fcc4ef None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:22.100953 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1f3f7d-b4de-4270-a411-fbf6b8fcc4ef None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:22.100953 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1f3f7d-b4de-4270-a411-fbf6b8fcc4ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:22.119540 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-de1f3f7d-b4de-4270-a411-fbf6b8fcc4ef None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:37:22.120763 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-de1f3f7d-b4de-4270-a411-fbf6b8fcc4ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:37:22.150144 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:22.190921 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 626/1289] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:37:22 2022] POST /identity/v3/auth/tokens => generated 2123 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:37:22.471554 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0f6aff-456b-4a0a-a198-8d1bd69c27f2 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:22.576374 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0f6aff-456b-4a0a-a198-8d1bd69c27f2 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:22.587592 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:22.614210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4186c99e-4323-43f9-8460-37248fc27846 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:22.615074 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4186c99e-4323-43f9-8460-37248fc27846 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:22.615074 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4186c99e-4323-43f9-8460-37248fc27846 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:22.615596 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4186c99e-4323-43f9-8460-37248fc27846 None admin] PATH_INFO: `/v3/services/placement` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:22.616076 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4186c99e-4323-43f9-8460-37248fc27846 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:22.617673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4186c99e-4323-43f9-8460-37248fc27846 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:22.628676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-4186c99e-4323-43f9-8460-37248fc27846 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 24 15:37:22.628676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:37:22.628676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 15:37:22.628676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 15:37:22.628676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 15:37:22.628676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 15:37:22.628676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 15:37:22.628676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:37:22.628676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 24 15:37:22.628676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:37:22.628676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 15:37:22.628676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:37:22.628676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:37:22.628676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:37:22.628676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:37:22.628676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:37:22.628676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:37:22.629934 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:37:22.629934 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:37:22.629934 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:37:22.629934 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:37:22.629934 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:37:22.629934 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:37:22.629934 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 15:37:22.629934 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 15:37:22.629934 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 15:37:22.629934 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 15:37:22.629934 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:37:22.629934 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:37:22.629934 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:37:22.629934 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:37:22.629934 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:37:22.629934 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:37:22.630785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:37:22.630785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:37:22.630785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:37:22.630785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:37:22.630785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:37:22.630785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:37:22.630785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:37:22.630785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:37:22.630785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:37:22.630785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:37:22.630785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 15:37:22.630785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:37:22.630785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 24 15:37:22.630785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:37:22.630785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 664/1290] 10.222.0.102 () {56 vars in 1122 bytes} [Wed Aug 24 15:37:22 2022] GET /identity/v3/services/placement => generated 90 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:37:22.635925 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1f3f7d-b4de-4270-a411-fbf6b8fcc4ef None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:22.737724 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1f3f7d-b4de-4270-a411-fbf6b8fcc4ef None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:22.772504 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a58d877-90c8-49d0-b0e8-5c4d18f3173a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:22.773104 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a58d877-90c8-49d0-b0e8-5c4d18f3173a None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:22.773379 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a58d877-90c8-49d0-b0e8-5c4d18f3173a None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:22.773648 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a58d877-90c8-49d0-b0e8-5c4d18f3173a None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:22.774308 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a58d877-90c8-49d0-b0e8-5c4d18f3173a None admin] RBAC: Adding query filter params (name=placement) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:37:22.774571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a58d877-90c8-49d0-b0e8-5c4d18f3173a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:22.775955 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a58d877-90c8-49d0-b0e8-5c4d18f3173a None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:22.781183 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 627/1291] 10.222.0.102 () {56 vars in 1121 bytes} [Wed Aug 24 15:37:22 2022] GET /identity/v3/services?name=placement => generated 127 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:37:22.786955 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4186c99e-4323-43f9-8460-37248fc27846 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:22.889145 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4186c99e-4323-43f9-8460-37248fc27846 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:22.924892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8989e613-dee6-4dc7-8fa1-213ca7c73436 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:22.925507 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8989e613-dee6-4dc7-8fa1-213ca7c73436 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:22.925788 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8989e613-dee6-4dc7-8fa1-213ca7c73436 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:22.926066 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8989e613-dee6-4dc7-8fa1-213ca7c73436 None admin] PATH_INFO: `/v3/services` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:22.926701 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8989e613-dee6-4dc7-8fa1-213ca7c73436 None admin] RBAC: Adding query filter params (type=placement) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:37:22.927004 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8989e613-dee6-4dc7-8fa1-213ca7c73436 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:22.928671 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8989e613-dee6-4dc7-8fa1-213ca7c73436 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:22.933068 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 665/1292] 10.222.0.102 () {56 vars in 1121 bytes} [Wed Aug 24 15:37:22 2022] GET /identity/v3/services?type=placement => generated 127 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:37:24.840336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a23150-940a-4107-be38-fba2abb99639 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:24.840894 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a23150-940a-4107-be38-fba2abb99639 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:24.841093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a23150-940a-4107-be38-fba2abb99639 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:24.842886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 628/1293] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:37:24 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:37:24.848521 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ec9cc5-b4ef-4e61-84c7-7c6c9c701c8f None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:24.848935 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ec9cc5-b4ef-4e61-84c7-7c6c9c701c8f None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:24.849055 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ec9cc5-b4ef-4e61-84c7-7c6c9c701c8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:24.867894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-72ec9cc5-b4ef-4e61-84c7-7c6c9c701c8f None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:37:24.869116 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-72ec9cc5-b4ef-4e61-84c7-7c6c9c701c8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:37:24.900029 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:24.940737 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 666/1294] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:37:24 2022] POST /identity/v3/auth/tokens => generated 2123 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:37:25.253717 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a23150-940a-4107-be38-fba2abb99639 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:25.286678 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:25.359495 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a23150-940a-4107-be38-fba2abb99639 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:25.398335 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6200451-58a5-48ad-a638-38f63f7ee0fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:25.398941 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6200451-58a5-48ad-a638-38f63f7ee0fe None admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:25.399223 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6200451-58a5-48ad-a638-38f63f7ee0fe None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:25.399498 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6200451-58a5-48ad-a638-38f63f7ee0fe None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:25.400187 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6200451-58a5-48ad-a638-38f63f7ee0fe None admin] RBAC: Authorizing `identity:create_service()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:25.401558 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6200451-58a5-48ad-a638-38f63f7ee0fe None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:25.416711 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 629/1295] 10.222.0.102 () {60 vars in 1146 bytes} [Wed Aug 24 15:37:25 2022] POST /identity/v3/services => generated 247 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:37:27.331043 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba839da-0ce2-48fb-8d7a-eee0a0c0b8e4 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:27.331572 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba839da-0ce2-48fb-8d7a-eee0a0c0b8e4 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:27.331572 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba839da-0ce2-48fb-8d7a-eee0a0c0b8e4 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:27.332974 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 667/1296] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:37:27 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:37:27.338126 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d9a80e-52df-4790-a01c-b29776fae353 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:27.338546 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d9a80e-52df-4790-a01c-b29776fae353 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:27.338678 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d9a80e-52df-4790-a01c-b29776fae353 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:27.357202 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-11d9a80e-52df-4790-a01c-b29776fae353 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:37:27.359041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-11d9a80e-52df-4790-a01c-b29776fae353 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:37:27.434750 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 630/1297] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:37:27 2022] POST /identity/v3/auth/tokens => generated 2226 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:37:27.707449 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba839da-0ce2-48fb-8d7a-eee0a0c0b8e4 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:27.821551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba839da-0ce2-48fb-8d7a-eee0a0c0b8e4 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:27.862591 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaff9892-2bb2-4de7-a6fc-a1f6909cb577 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:27.863225 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaff9892-2bb2-4de7-a6fc-a1f6909cb577 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:27.863510 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaff9892-2bb2-4de7-a6fc-a1f6909cb577 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:27.863815 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaff9892-2bb2-4de7-a6fc-a1f6909cb577 None admin] PATH_INFO: `/v3/services/placement` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:27.864452 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaff9892-2bb2-4de7-a6fc-a1f6909cb577 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:27.865785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaff9892-2bb2-4de7-a6fc-a1f6909cb577 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:27.874625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-aaff9892-2bb2-4de7-a6fc-a1f6909cb577 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 24 15:37:27.874625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:37:27.874625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 15:37:27.874625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 15:37:27.874625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 15:37:27.874625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 15:37:27.874625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 15:37:27.874625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:37:27.874625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 24 15:37:27.874625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:37:27.874625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 15:37:27.874625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:37:27.874625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:37:27.874625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:37:27.874625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:37:27.874625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:37:27.874625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:37:27.875836 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:37:27.875836 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:37:27.875836 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:37:27.875836 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:37:27.875836 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:37:27.875836 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:37:27.875836 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 15:37:27.875836 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 15:37:27.875836 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 15:37:27.875836 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 15:37:27.875836 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:37:27.875836 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:37:27.875836 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:37:27.875836 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:37:27.875836 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:37:27.875836 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:37:27.876794 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:37:27.876794 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:37:27.876794 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:37:27.876794 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:37:27.876794 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:37:27.876794 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:37:27.876794 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:37:27.876794 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:37:27.876794 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:37:27.876794 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:37:27.876794 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 15:37:27.876794 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:37:27.876794 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 24 15:37:27.876794 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:37:27.876794 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 668/1298] 10.222.0.102 () {56 vars in 1122 bytes} [Wed Aug 24 15:37:27 2022] GET /identity/v3/services/placement => generated 90 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:37:27.881780 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d9a80e-52df-4790-a01c-b29776fae353 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:27.900461 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:27.984253 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d9a80e-52df-4790-a01c-b29776fae353 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:28.018178 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f65a5d3-101a-4230-9d34-a13f66248cc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:28.018768 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f65a5d3-101a-4230-9d34-a13f66248cc1 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:28.019042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f65a5d3-101a-4230-9d34-a13f66248cc1 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:28.019313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f65a5d3-101a-4230-9d34-a13f66248cc1 None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:28.019982 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f65a5d3-101a-4230-9d34-a13f66248cc1 None admin] RBAC: Adding query filter params (name=placement) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:37:28.020283 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f65a5d3-101a-4230-9d34-a13f66248cc1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:28.021777 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f65a5d3-101a-4230-9d34-a13f66248cc1 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:28.027084 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 631/1299] 10.222.0.102 () {56 vars in 1121 bytes} [Wed Aug 24 15:37:27 2022] GET /identity/v3/services?name=placement => generated 360 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:37:28.034959 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaff9892-2bb2-4de7-a6fc-a1f6909cb577 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:28.072282 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:28.150789 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaff9892-2bb2-4de7-a6fc-a1f6909cb577 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:28.183914 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9867b5d0-2967-4283-93fa-dd0e81ea7f68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:28.184516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9867b5d0-2967-4283-93fa-dd0e81ea7f68 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:28.184788 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9867b5d0-2967-4283-93fa-dd0e81ea7f68 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:28.185055 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9867b5d0-2967-4283-93fa-dd0e81ea7f68 None admin] PATH_INFO: `/v3/endpoints` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:28.186274 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9867b5d0-2967-4283-93fa-dd0e81ea7f68 None admin] RBAC: Adding query filter params (interface=public, service_id=e4c30845140244ba8d4564de0b73d96a, region_id=RegionOne) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:37:28.186575 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9867b5d0-2967-4283-93fa-dd0e81ea7f68 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:28.187958 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9867b5d0-2967-4283-93fa-dd0e81ea7f68 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:28.194905 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 669/1300] 10.222.0.102 () {56 vars in 1256 bytes} [Wed Aug 24 15:37:28 2022] GET /identity/v3/endpoints?service_id=e4c30845140244ba8d4564de0b73d96a&interface=public®ion_id=RegionOne => generated 195 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:37:28.200792 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f65a5d3-101a-4230-9d34-a13f66248cc1 None admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:28.284868 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:28.303192 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f65a5d3-101a-4230-9d34-a13f66248cc1 None admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:28.337802 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd434e9-93ec-4957-a449-dac6a5afb605 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:28.338400 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd434e9-93ec-4957-a449-dac6a5afb605 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:28.338921 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd434e9-93ec-4957-a449-dac6a5afb605 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:28.339036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd434e9-93ec-4957-a449-dac6a5afb605 None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:28.339544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd434e9-93ec-4957-a449-dac6a5afb605 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:28.340932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd434e9-93ec-4957-a449-dac6a5afb605 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:28.348220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 632/1301] 10.222.0.102 () {56 vars in 1092 bytes} [Wed Aug 24 15:37:28 2022] GET /identity/v3/services => generated 2201 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:37:30.263591 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b5d993-ba11-4bea-aed6-8f5dce231f65 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:30.264186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b5d993-ba11-4bea-aed6-8f5dce231f65 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:30.264292 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b5d993-ba11-4bea-aed6-8f5dce231f65 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:30.265572 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 670/1302] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:37:30 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:37:30.271253 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ad899f-d9ee-47dd-83a5-e886e1613c92 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:30.271716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ad899f-d9ee-47dd-83a5-e886e1613c92 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:30.271835 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ad899f-d9ee-47dd-83a5-e886e1613c92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:30.290271 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-25ad899f-d9ee-47dd-83a5-e886e1613c92 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:37:30.291459 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-25ad899f-d9ee-47dd-83a5-e886e1613c92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:37:30.358450 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 633/1303] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:37:30 2022] POST /identity/v3/auth/tokens => generated 2226 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 15:37:30.623212 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b5d993-ba11-4bea-aed6-8f5dce231f65 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:30.724471 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:30.731576 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b5d993-ba11-4bea-aed6-8f5dce231f65 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:30.766768 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffc5d33-265d-4266-b478-4b22b20ab086 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:30.767395 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffc5d33-265d-4266-b478-4b22b20ab086 None admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:30.767713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffc5d33-265d-4266-b478-4b22b20ab086 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:30.767994 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffc5d33-265d-4266-b478-4b22b20ab086 None admin] PATH_INFO: `/v3/services/placement` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:30.768576 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffc5d33-265d-4266-b478-4b22b20ab086 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:30.769949 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffc5d33-265d-4266-b478-4b22b20ab086 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:30.778810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application [None req-6ffc5d33-265d-4266-b478-4b22b20ab086 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 24 15:37:30.778810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:37:30.778810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 15:37:30.778810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 15:37:30.778810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 15:37:30.778810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 15:37:30.778810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 15:37:30.778810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:37:30.778810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 24 15:37:30.778810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:37:30.778810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 15:37:30.778810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:37:30.778810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 15:37:30.778810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 15:37:30.778810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 15:37:30.778810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 15:37:30.778810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 15:37:30.780621 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 15:37:30.780621 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 15:37:30.780621 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 15:37:30.780621 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 15:37:30.780621 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 15:37:30.780621 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 15:37:30.780621 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 15:37:30.780621 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 15:37:30.780621 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 15:37:30.780621 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 15:37:30.780621 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 15:37:30.780621 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 15:37:30.780621 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 15:37:30.780621 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 15:37:30.780621 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 15:37:30.780621 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 15:37:30.781550 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 15:37:30.781550 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application with Lock( Aug 24 15:37:30.781550 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 15:37:30.781550 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self._enter() Aug 24 15:37:30.781550 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 15:37:30.781550 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 15:37:30.781550 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 15:37:30.781550 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application return self.creator() Aug 24 15:37:30.781550 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 15:37:30.781550 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application created_value = creator( Aug 24 15:37:30.781550 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 15:37:30.781550 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 15:37:30.781550 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 24 15:37:30.781550 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: ERROR keystone.server.flask.application Aug 24 15:37:30.781550 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 671/1304] 10.222.0.102 () {56 vars in 1122 bytes} [Wed Aug 24 15:37:30 2022] GET /identity/v3/services/placement => generated 90 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 15:37:30.787280 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ad899f-d9ee-47dd-83a5-e886e1613c92 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:30.865378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:30.890032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ad899f-d9ee-47dd-83a5-e886e1613c92 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:30.926487 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b878edaf-d3b9-45c1-81d0-a19874a0aa62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:30.927346 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b878edaf-d3b9-45c1-81d0-a19874a0aa62 None admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:30.927644 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b878edaf-d3b9-45c1-81d0-a19874a0aa62 None admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:30.927927 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b878edaf-d3b9-45c1-81d0-a19874a0aa62 None admin] PATH_INFO: `/v3/services` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:30.928563 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b878edaf-d3b9-45c1-81d0-a19874a0aa62 None admin] RBAC: Adding query filter params (name=placement) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:37:30.928869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b878edaf-d3b9-45c1-81d0-a19874a0aa62 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:30.930239 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b878edaf-d3b9-45c1-81d0-a19874a0aa62 None admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:30.935265 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 634/1305] 10.222.0.102 () {56 vars in 1121 bytes} [Wed Aug 24 15:37:30 2022] GET /identity/v3/services?name=placement => generated 360 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:37:30.941055 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffc5d33-265d-4266-b478-4b22b20ab086 None admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:37:31.041282 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffc5d33-265d-4266-b478-4b22b20ab086 None admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:37:31.075037 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9116cb87-b45d-46a9-99b0-b7406710f887 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:37:31.075657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9116cb87-b45d-46a9-99b0-b7406710f887 None admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:37:31.075969 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9116cb87-b45d-46a9-99b0-b7406710f887 None admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:37:31.076240 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9116cb87-b45d-46a9-99b0-b7406710f887 None admin] PATH_INFO: `/v3/endpoints` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:37:31.076983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9116cb87-b45d-46a9-99b0-b7406710f887 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:37:31.078349 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9116cb87-b45d-46a9-99b0-b7406710f887 None admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:37:31.099171 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 672/1306] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:37:30 2022] POST /identity/v3/endpoints => generated 333 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:37:40.932333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=30 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:37:41.094377 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:38:37.744530 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da09f3b-d3b6-418c-b675-8cbb1a55f765 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:38:37.745075 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da09f3b-d3b6-418c-b675-8cbb1a55f765 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:38:37.745075 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da09f3b-d3b6-418c-b675-8cbb1a55f765 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:38:37.751741 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38985d57-37ce-47e7-bbb7-24b936777048 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:38:37.752227 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38985d57-37ce-47e7-bbb7-24b936777048 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:38:37.752312 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38985d57-37ce-47e7-bbb7-24b936777048 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:38:37.757786 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 635/1307] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:38:37 2022] GET /identity => generated 271 bytes in 15 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:38:37.877175 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-38985d57-37ce-47e7-bbb7-24b936777048 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:38:37.880507 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-38985d57-37ce-47e7-bbb7-24b936777048 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:38:38.025350 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 673/1308] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:38:37 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 274 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:38:38.361072 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0434446-62de-4f71-9d79-930d25ba6b8c None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:38:38.361477 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0434446-62de-4f71-9d79-930d25ba6b8c None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:38:38.361695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0434446-62de-4f71-9d79-930d25ba6b8c None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:38:38.363375 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 636/1309] 10.222.0.102 () {54 vars in 970 bytes} [Wed Aug 24 15:38:38 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:38:38.371009 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf35e37-28be-4086-a93c-b23d0998e5fc None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:38:38.371315 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf35e37-28be-4086-a93c-b23d0998e5fc None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:38:38.371732 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf35e37-28be-4086-a93c-b23d0998e5fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:38:38.464753 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-faf35e37-28be-4086-a93c-b23d0998e5fc None None] MFA Rules not processed for user `48939674c1fd4cf599691d510fcd8858`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:38:38.465987 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-faf35e37-28be-4086-a93c-b23d0998e5fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:38:38.491624 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:38:38.557300 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 674/1310] 10.222.0.102 () {58 vars in 1067 bytes} [Wed Aug 24 15:38:38 2022] POST /identity/v3/auth/tokens => generated 3131 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:38:38.566494 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0434446-62de-4f71-9d79-930d25ba6b8c None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:38:38.568384 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-e0434446-62de-4f71-9d79-930d25ba6b8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:38:38.700869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0434446-62de-4f71-9d79-930d25ba6b8c None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:38:38.785550 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9c4609-b7f8-4180-a6af-af093ff551a6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:38:38.786282 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9c4609-b7f8-4180-a6af-af093ff551a6 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:38:38.786282 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9c4609-b7f8-4180-a6af-af093ff551a6 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:38:38.786457 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9c4609-b7f8-4180-a6af-af093ff551a6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:38:38.787993 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-7d9c4609-b7f8-4180-a6af-af093ff551a6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:38:38.862460 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:38:38.924381 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d9c4609-b7f8-4180-a6af-af093ff551a6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:38:38.925830 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d9c4609-b7f8-4180-a6af-af093ff551a6 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:38:38.947111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 637/1311] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:38:38 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:38:41.228834 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dae0bdd-fa7a-451d-ab4e-724a98afa5a3 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:38:41.229362 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dae0bdd-fa7a-451d-ab4e-724a98afa5a3 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:38:41.229442 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dae0bdd-fa7a-451d-ab4e-724a98afa5a3 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:38:41.230936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 675/1312] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:38:41 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:38:41.236361 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a91ba1f-a3fd-4e6b-88b6-3206bbe78ccd None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:38:41.236844 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a91ba1f-a3fd-4e6b-88b6-3206bbe78ccd None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:38:41.236924 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a91ba1f-a3fd-4e6b-88b6-3206bbe78ccd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:38:41.340111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-3a91ba1f-a3fd-4e6b-88b6-3206bbe78ccd None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:38:41.341359 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-3a91ba1f-a3fd-4e6b-88b6-3206bbe78ccd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:38:41.443079 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 638/1313] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:38:41 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:38:41.744758 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e7f6b0-d7db-46e5-855e-12c8e494544e None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:38:41.745156 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e7f6b0-d7db-46e5-855e-12c8e494544e None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:38:41.745502 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e7f6b0-d7db-46e5-855e-12c8e494544e None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:38:41.746972 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 676/1314] 10.222.0.102 () {54 vars in 970 bytes} [Wed Aug 24 15:38:41 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:38:41.753185 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0233cc2a-1f24-4956-995f-60fa321a9d66 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:38:41.753468 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0233cc2a-1f24-4956-995f-60fa321a9d66 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:38:41.753745 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0233cc2a-1f24-4956-995f-60fa321a9d66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:38:41.766114 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:38:41.840590 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-0233cc2a-1f24-4956-995f-60fa321a9d66 None None] MFA Rules not processed for user `48939674c1fd4cf599691d510fcd8858`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:38:41.841991 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-0233cc2a-1f24-4956-995f-60fa321a9d66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:38:41.931738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 639/1315] 10.222.0.102 () {58 vars in 1067 bytes} [Wed Aug 24 15:38:41 2022] POST /identity/v3/auth/tokens => generated 3131 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:38:41.940694 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e7f6b0-d7db-46e5-855e-12c8e494544e None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:38:41.942308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-e3e7f6b0-d7db-46e5-855e-12c8e494544e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:38:42.052425 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e7f6b0-d7db-46e5-855e-12c8e494544e None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:38:42.130134 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:38:42.133685 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94c3677-5280-4558-b36d-52ed0df4fec1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:38:42.134494 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94c3677-5280-4558-b36d-52ed0df4fec1 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:38:42.135170 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94c3677-5280-4558-b36d-52ed0df4fec1 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:38:42.135486 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94c3677-5280-4558-b36d-52ed0df4fec1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:38:42.137333 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-e94c3677-5280-4558-b36d-52ed0df4fec1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:38:42.248346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e94c3677-5280-4558-b36d-52ed0df4fec1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:38:42.249793 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e94c3677-5280-4558-b36d-52ed0df4fec1 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:38:42.270508 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 677/1316] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:38:41 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:38:44.783502 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88be7c69-55d3-401f-8165-cb0751d8c550 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:38:44.784108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88be7c69-55d3-401f-8165-cb0751d8c550 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:38:44.784108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88be7c69-55d3-401f-8165-cb0751d8c550 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:38:44.785403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 640/1317] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:38:44 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:38:44.791964 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230e2726-a5c9-4ee8-9c8b-f40174599789 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:38:44.792495 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230e2726-a5c9-4ee8-9c8b-f40174599789 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:38:44.792614 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230e2726-a5c9-4ee8-9c8b-f40174599789 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:38:44.862721 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-230e2726-a5c9-4ee8-9c8b-f40174599789 None None] MFA Rules not processed for user `cb1c2ffc97364fa48e0d109484b7871c`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:38:44.863757 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-230e2726-a5c9-4ee8-9c8b-f40174599789 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:38:44.917838 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:38:44.963533 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 678/1318] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:38:44 2022] POST /identity/v3/auth/tokens => generated 3253 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:38:45.255790 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88be7c69-55d3-401f-8165-cb0751d8c550 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:38:45.257371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-88be7c69-55d3-401f-8165-cb0751d8c550 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:38:45.361896 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88be7c69-55d3-401f-8165-cb0751d8c550 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:38:45.364712 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:38:45.448220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219eae94-217d-4a3d-9ec3-d41e51d2c89e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:38:45.448927 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219eae94-217d-4a3d-9ec3-d41e51d2c89e service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:38:45.449205 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219eae94-217d-4a3d-9ec3-d41e51d2c89e service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:38:45.449474 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219eae94-217d-4a3d-9ec3-d41e51d2c89e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:38:45.451129 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-219eae94-217d-4a3d-9ec3-d41e51d2c89e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:38:45.581066 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-219eae94-217d-4a3d-9ec3-d41e51d2c89e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:38:45.582420 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-219eae94-217d-4a3d-9ec3-d41e51d2c89e service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:38:45.601872 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 641/1319] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:38:45 2022] GET /identity/v3/auth/tokens => generated 3253 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:38:48.264277 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf69d77-5ca0-4c34-9c4f-d2096c7b69b7 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:38:48.264840 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf69d77-5ca0-4c34-9c4f-d2096c7b69b7 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:38:48.264840 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf69d77-5ca0-4c34-9c4f-d2096c7b69b7 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:38:48.266188 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 679/1320] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:38:48 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:38:48.271495 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05312fb-8b72-49a3-8a28-0948b91bbb9b None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:38:48.271965 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05312fb-8b72-49a3-8a28-0948b91bbb9b None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:38:48.272077 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05312fb-8b72-49a3-8a28-0948b91bbb9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:38:48.313532 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:38:48.340858 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-c05312fb-8b72-49a3-8a28-0948b91bbb9b None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:38:48.342084 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-c05312fb-8b72-49a3-8a28-0948b91bbb9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:38:48.453524 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 642/1321] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:38:48 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:38:48.740172 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf69d77-5ca0-4c34-9c4f-d2096c7b69b7 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:38:48.741729 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-9cf69d77-5ca0-4c34-9c4f-d2096c7b69b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:38:48.853319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf69d77-5ca0-4c34-9c4f-d2096c7b69b7 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:38:48.927867 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e29ee39f-0465-4dfa-ac11-b000d555af82 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:38:48.928473 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29ee39f-0465-4dfa-ac11-b000d555af82 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:38:48.928755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29ee39f-0465-4dfa-ac11-b000d555af82 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:38:48.929030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29ee39f-0465-4dfa-ac11-b000d555af82 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:38:48.930823 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-e29ee39f-0465-4dfa-ac11-b000d555af82 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:38:48.938415 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:38:49.053312 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e29ee39f-0465-4dfa-ac11-b000d555af82 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:38:49.054727 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e29ee39f-0465-4dfa-ac11-b000d555af82 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:38:49.074467 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 680/1322] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:38:48 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:38:51.738823 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43229d62-e7b0-4499-8d20-a41baaea2c83 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:38:51.739500 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43229d62-e7b0-4499-8d20-a41baaea2c83 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:38:51.739500 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43229d62-e7b0-4499-8d20-a41baaea2c83 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:38:51.740745 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 643/1323] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:38:51 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:38:51.746315 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf032be-d734-462a-ad5c-bd931406fb0d None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:38:51.746851 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf032be-d734-462a-ad5c-bd931406fb0d None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:38:51.746957 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf032be-d734-462a-ad5c-bd931406fb0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:38:51.819590 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-abf032be-d734-462a-ad5c-bd931406fb0d None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:38:51.821163 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-abf032be-d734-462a-ad5c-bd931406fb0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:38:51.884194 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:38:51.928114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 681/1324] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:38:51 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:38:52.218281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43229d62-e7b0-4499-8d20-a41baaea2c83 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:38:52.219830 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-43229d62-e7b0-4499-8d20-a41baaea2c83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:38:52.331654 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43229d62-e7b0-4499-8d20-a41baaea2c83 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:38:52.344900 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:38:52.425506 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1c7369-38ec-4723-a087-9690135de1b5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:38:52.426115 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1c7369-38ec-4723-a087-9690135de1b5 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:38:52.426392 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1c7369-38ec-4723-a087-9690135de1b5 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:38:52.426797 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1c7369-38ec-4723-a087-9690135de1b5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:38:52.430351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-7a1c7369-38ec-4723-a087-9690135de1b5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:38:52.546392 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a1c7369-38ec-4723-a087-9690135de1b5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:38:52.547749 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a1c7369-38ec-4723-a087-9690135de1b5 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:38:52.567043 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 644/1325] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:38:52 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:38:54.650728 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9712de-fe53-4165-9162-4b24422219da None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:38:54.651265 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9712de-fe53-4165-9162-4b24422219da None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:38:54.651265 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9712de-fe53-4165-9162-4b24422219da None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:38:54.652708 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 682/1326] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:38:54 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:38:54.658037 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af51719-54a6-4594-a8d6-29f9c080b095 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:38:54.658487 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af51719-54a6-4594-a8d6-29f9c080b095 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:38:54.658783 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af51719-54a6-4594-a8d6-29f9c080b095 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:38:54.711277 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:38:54.732579 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-0af51719-54a6-4594-a8d6-29f9c080b095 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:38:54.733912 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-0af51719-54a6-4594-a8d6-29f9c080b095 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:38:54.858201 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 645/1327] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:38:54 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:38:55.147624 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9712de-fe53-4165-9162-4b24422219da None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:38:55.149863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-9d9712de-fe53-4165-9162-4b24422219da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:38:55.271132 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9712de-fe53-4165-9162-4b24422219da None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:38:55.352471 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dadf096-1792-4207-9a00-e70cad57f3c2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:38:55.353080 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dadf096-1792-4207-9a00-e70cad57f3c2 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:38:55.353366 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dadf096-1792-4207-9a00-e70cad57f3c2 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:38:55.353637 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dadf096-1792-4207-9a00-e70cad57f3c2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:38:55.355277 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-6dadf096-1792-4207-9a00-e70cad57f3c2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:38:55.373518 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:38:55.471173 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dadf096-1792-4207-9a00-e70cad57f3c2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:38:55.472573 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dadf096-1792-4207-9a00-e70cad57f3c2 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:38:55.490866 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 683/1328] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:38:55 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:38:57.517207 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658d13e7-c487-4dee-88c9-9f63be1417ac None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:38:57.517767 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658d13e7-c487-4dee-88c9-9f63be1417ac None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:38:57.517879 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658d13e7-c487-4dee-88c9-9f63be1417ac None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:38:57.519191 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 646/1329] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:38:57 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:38:57.525096 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f6191f-429c-473a-a2d5-10a6c8f145ab None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:38:57.525855 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f6191f-429c-473a-a2d5-10a6c8f145ab None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:38:57.526137 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f6191f-429c-473a-a2d5-10a6c8f145ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:38:57.596689 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-09f6191f-429c-473a-a2d5-10a6c8f145ab None None] MFA Rules not processed for user `cb1c2ffc97364fa48e0d109484b7871c`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:38:57.597921 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-09f6191f-429c-473a-a2d5-10a6c8f145ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:38:57.665328 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:38:57.701903 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 684/1330] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:38:57 2022] POST /identity/v3/auth/tokens => generated 3253 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:38:57.988679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658d13e7-c487-4dee-88c9-9f63be1417ac None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:38:57.990256 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-658d13e7-c487-4dee-88c9-9f63be1417ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:38:58.099165 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658d13e7-c487-4dee-88c9-9f63be1417ac None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:38:58.121566 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:38:58.177782 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2a4cd5-113b-43cb-bb65-a7d7628cb52c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:38:58.178302 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2a4cd5-113b-43cb-bb65-a7d7628cb52c service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:38:58.178576 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2a4cd5-113b-43cb-bb65-a7d7628cb52c service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:38:58.178845 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2a4cd5-113b-43cb-bb65-a7d7628cb52c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:38:58.181043 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-3c2a4cd5-113b-43cb-bb65-a7d7628cb52c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:38:58.311318 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2a4cd5-113b-43cb-bb65-a7d7628cb52c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:38:58.312792 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2a4cd5-113b-43cb-bb65-a7d7628cb52c service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:38:58.332566 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 647/1331] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:38:57 2022] GET /identity/v3/auth/tokens => generated 3253 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:39:00.565342 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a82308-3a0f-4bc7-afd7-cb9654563774 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:00.565882 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a82308-3a0f-4bc7-afd7-cb9654563774 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:00.565882 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a82308-3a0f-4bc7-afd7-cb9654563774 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:00.567248 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 685/1332] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:39:00 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:39:00.573411 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04278366-a7e2-42c4-840a-f8787cf758f8 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:00.573871 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04278366-a7e2-42c4-840a-f8787cf758f8 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:00.573985 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04278366-a7e2-42c4-840a-f8787cf758f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:00.636851 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:00.639864 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-04278366-a7e2-42c4-840a-f8787cf758f8 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:39:00.641829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-04278366-a7e2-42c4-840a-f8787cf758f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:00.741299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 648/1333] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:39:00 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:39:01.028089 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a82308-3a0f-4bc7-afd7-cb9654563774 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:39:01.029776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-99a82308-3a0f-4bc7-afd7-cb9654563774 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:01.143873 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a82308-3a0f-4bc7-afd7-cb9654563774 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:39:01.221130 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf03314-4592-4958-9a5a-becd01052271 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:39:01.221728 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf03314-4592-4958-9a5a-becd01052271 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:01.222008 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf03314-4592-4958-9a5a-becd01052271 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:01.222588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf03314-4592-4958-9a5a-becd01052271 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:01.223914 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-eaf03314-4592-4958-9a5a-becd01052271 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:01.252951 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:01.359808 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf03314-4592-4958-9a5a-becd01052271 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:39:01.361452 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf03314-4592-4958-9a5a-becd01052271 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:39:01.382695 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 686/1334] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:39:01 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:39:03.410085 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc668ba8-3244-43d1-9c8c-b67fc0807e71 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:03.410614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc668ba8-3244-43d1-9c8c-b67fc0807e71 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:03.410706 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc668ba8-3244-43d1-9c8c-b67fc0807e71 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:03.412041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 649/1335] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:39:03 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:39:03.417816 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895941ff-2284-4f45-ac48-c18f1c941b12 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:03.418507 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895941ff-2284-4f45-ac48-c18f1c941b12 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:03.418507 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895941ff-2284-4f45-ac48-c18f1c941b12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:03.494017 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-895941ff-2284-4f45-ac48-c18f1c941b12 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:39:03.495342 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-895941ff-2284-4f45-ac48-c18f1c941b12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:03.594678 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:03.626056 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 687/1336] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:39:03 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:39:03.997213 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc668ba8-3244-43d1-9c8c-b67fc0807e71 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:39:03.998770 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-fc668ba8-3244-43d1-9c8c-b67fc0807e71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:04.137714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc668ba8-3244-43d1-9c8c-b67fc0807e71 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:39:04.169804 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:04.222236 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26834365-bc8c-4989-9f68-4cf89d8c80f4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:39:04.223046 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26834365-bc8c-4989-9f68-4cf89d8c80f4 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:04.223333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26834365-bc8c-4989-9f68-4cf89d8c80f4 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:04.223602 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26834365-bc8c-4989-9f68-4cf89d8c80f4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:04.225264 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-26834365-bc8c-4989-9f68-4cf89d8c80f4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:04.345499 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26834365-bc8c-4989-9f68-4cf89d8c80f4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:39:04.347059 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26834365-bc8c-4989-9f68-4cf89d8c80f4 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:39:04.385347 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 650/1337] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:39:03 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:39:06.758707 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9697054-8897-4c42-8a95-034e4acc36f5 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:06.759295 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9697054-8897-4c42-8a95-034e4acc36f5 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:06.759295 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9697054-8897-4c42-8a95-034e4acc36f5 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:06.760657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 688/1338] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:39:06 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:39:06.766062 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2351653-2bd4-4e34-bd75-73f63f7feec0 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:06.766512 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2351653-2bd4-4e34-bd75-73f63f7feec0 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:06.766633 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2351653-2bd4-4e34-bd75-73f63f7feec0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:06.840937 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-a2351653-2bd4-4e34-bd75-73f63f7feec0 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:39:06.842344 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-a2351653-2bd4-4e34-bd75-73f63f7feec0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:06.851294 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:06.942878 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 651/1339] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:39:06 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:39:07.225815 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9697054-8897-4c42-8a95-034e4acc36f5 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:39:07.228170 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-c9697054-8897-4c42-8a95-034e4acc36f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:07.338608 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9697054-8897-4c42-8a95-034e4acc36f5 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:39:07.412765 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c653769-b9fd-48ed-a9c0-0bf0286afd8d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:39:07.414016 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c653769-b9fd-48ed-a9c0-0bf0286afd8d service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:07.414296 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c653769-b9fd-48ed-a9c0-0bf0286afd8d service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:07.414569 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c653769-b9fd-48ed-a9c0-0bf0286afd8d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:07.416314 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-3c653769-b9fd-48ed-a9c0-0bf0286afd8d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:07.453162 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:07.528630 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c653769-b9fd-48ed-a9c0-0bf0286afd8d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:39:07.529990 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c653769-b9fd-48ed-a9c0-0bf0286afd8d service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:39:07.548626 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 689/1340] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:39:07 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:39:11.543875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa04652-cba3-4f02-8091-5ae13d3b7f50 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:11.544410 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa04652-cba3-4f02-8091-5ae13d3b7f50 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:11.544410 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa04652-cba3-4f02-8091-5ae13d3b7f50 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:11.545768 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 652/1341] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:39:11 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:39:11.551343 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9af71e-dca1-46e5-b13f-05ef2afe7b54 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:11.551826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9af71e-dca1-46e5-b13f-05ef2afe7b54 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:11.551933 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9af71e-dca1-46e5-b13f-05ef2afe7b54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:11.621321 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-8c9af71e-dca1-46e5-b13f-05ef2afe7b54 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:39:11.622574 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-8c9af71e-dca1-46e5-b13f-05ef2afe7b54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:11.718770 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:11.733624 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 690/1342] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:39:11 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:39:12.015775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa04652-cba3-4f02-8091-5ae13d3b7f50 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:39:12.017322 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-8fa04652-cba3-4f02-8091-5ae13d3b7f50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:12.130461 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa04652-cba3-4f02-8091-5ae13d3b7f50 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:39:12.171093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:12.209451 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa2bfac-8af2-4edc-95a6-b554896ca711 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:39:12.210312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa2bfac-8af2-4edc-95a6-b554896ca711 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:12.210615 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa2bfac-8af2-4edc-95a6-b554896ca711 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:12.211016 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa2bfac-8af2-4edc-95a6-b554896ca711 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:12.212735 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-6fa2bfac-8af2-4edc-95a6-b554896ca711 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:12.327314 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa2bfac-8af2-4edc-95a6-b554896ca711 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:39:12.328829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa2bfac-8af2-4edc-95a6-b554896ca711 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:39:12.348053 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 653/1343] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:39:12 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:39:14.997582 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec32cdc-401d-4af3-b79d-e8e82f86abec None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:14.998122 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec32cdc-401d-4af3-b79d-e8e82f86abec None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:14.998122 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec32cdc-401d-4af3-b79d-e8e82f86abec None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:14.999546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 691/1344] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:39:14 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:39:15.005301 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fe3ebb-38d6-4b98-9c61-70281b28fc06 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:15.005826 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fe3ebb-38d6-4b98-9c61-70281b28fc06 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:15.005826 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fe3ebb-38d6-4b98-9c61-70281b28fc06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:15.088264 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-f2fe3ebb-38d6-4b98-9c61-70281b28fc06 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:39:15.090175 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-f2fe3ebb-38d6-4b98-9c61-70281b28fc06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:15.111837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:15.196669 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 654/1345] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:39:15 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:39:15.484219 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec32cdc-401d-4af3-b79d-e8e82f86abec None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:39:15.485799 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-7ec32cdc-401d-4af3-b79d-e8e82f86abec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:15.605062 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec32cdc-401d-4af3-b79d-e8e82f86abec None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:39:15.684868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14a362d-790f-43e7-90bd-be5dd98c8f9b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:39:15.685477 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14a362d-790f-43e7-90bd-be5dd98c8f9b service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:15.685751 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14a362d-790f-43e7-90bd-be5dd98c8f9b service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:15.686042 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14a362d-790f-43e7-90bd-be5dd98c8f9b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:15.687618 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-d14a362d-790f-43e7-90bd-be5dd98c8f9b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:15.735508 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:15.814368 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14a362d-790f-43e7-90bd-be5dd98c8f9b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:39:15.815853 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14a362d-790f-43e7-90bd-be5dd98c8f9b service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:39:15.838044 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 692/1346] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:39:15 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 354 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:39:20.367511 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9db60bf-4a56-4d36-a164-48a4e2ec688f None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:20.368020 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9db60bf-4a56-4d36-a164-48a4e2ec688f None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:20.368232 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9db60bf-4a56-4d36-a164-48a4e2ec688f None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:20.369622 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 655/1347] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:39:20 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:39:20.375462 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de01934c-7462-41fc-8cf0-5a416250a099 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:20.376052 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de01934c-7462-41fc-8cf0-5a416250a099 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:20.376052 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de01934c-7462-41fc-8cf0-5a416250a099 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:20.449528 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-de01934c-7462-41fc-8cf0-5a416250a099 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:39:20.450616 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-de01934c-7462-41fc-8cf0-5a416250a099 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:20.554297 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:20.557047 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 693/1348] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:39:20 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:39:20.842834 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9db60bf-4a56-4d36-a164-48a4e2ec688f None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:39:20.844447 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-d9db60bf-4a56-4d36-a164-48a4e2ec688f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:20.960238 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9db60bf-4a56-4d36-a164-48a4e2ec688f None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:39:21.011507 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:21.038730 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311da9f5-40f6-4fe8-8523-d776e9003dd6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:39:21.039391 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311da9f5-40f6-4fe8-8523-d776e9003dd6 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:21.039696 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311da9f5-40f6-4fe8-8523-d776e9003dd6 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:21.039973 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311da9f5-40f6-4fe8-8523-d776e9003dd6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:21.041595 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-311da9f5-40f6-4fe8-8523-d776e9003dd6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:21.175510 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311da9f5-40f6-4fe8-8523-d776e9003dd6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:39:21.178273 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311da9f5-40f6-4fe8-8523-d776e9003dd6 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:39:21.204150 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 656/1349] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:39:20 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:39:26.298709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896f86b9-74e1-44a4-a968-673c9dcab38a None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:26.300580 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896f86b9-74e1-44a4-a968-673c9dcab38a None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:26.301089 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896f86b9-74e1-44a4-a968-673c9dcab38a None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:26.303075 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 694/1350] 10.222.0.102 () {54 vars in 926 bytes} [Wed Aug 24 15:39:26 2022] GET /identity => generated 271 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:39:26.310967 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0547055-f1a4-4223-b729-ee5026c9e22c None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:26.310967 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0547055-f1a4-4223-b729-ee5026c9e22c None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:26.311578 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0547055-f1a4-4223-b729-ee5026c9e22c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:26.421064 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-a0547055-f1a4-4223-b729-ee5026c9e22c None None] MFA Rules not processed for user `0ee0002a76c3471f819b37fef510ef99`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:39:26.422273 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-a0547055-f1a4-4223-b729-ee5026c9e22c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:26.452733 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:26.608241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 657/1351] 10.222.0.102 () {58 vars in 1023 bytes} [Wed Aug 24 15:39:26 2022] POST /identity/v3/auth/tokens => generated 3318 bytes in 298 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:39:30.970997 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bf3a1f-65cc-48ae-b0d0-c773ffcec597 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:30.971548 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bf3a1f-65cc-48ae-b0d0-c773ffcec597 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:30.971913 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bf3a1f-65cc-48ae-b0d0-c773ffcec597 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:30.973532 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 695/1352] 10.222.0.102 () {54 vars in 926 bytes} [Wed Aug 24 15:39:30 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:39:30.978982 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb4d8ef-ce24-412f-a8a0-e18121dd5e43 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:30.979480 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb4d8ef-ce24-412f-a8a0-e18121dd5e43 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:30.979574 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb4d8ef-ce24-412f-a8a0-e18121dd5e43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:31.061642 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-bdb4d8ef-ce24-412f-a8a0-e18121dd5e43 None None] MFA Rules not processed for user `0ee0002a76c3471f819b37fef510ef99`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:39:31.062853 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-bdb4d8ef-ce24-412f-a8a0-e18121dd5e43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:31.143835 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:31.183032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 658/1353] 10.222.0.102 () {58 vars in 1023 bytes} [Wed Aug 24 15:39:30 2022] POST /identity/v3/auth/tokens => generated 3318 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:39:32.570685 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70891d88-83a8-4192-bcef-adcca1eead71 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:32.571212 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70891d88-83a8-4192-bcef-adcca1eead71 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:32.571212 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70891d88-83a8-4192-bcef-adcca1eead71 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:32.572617 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 696/1354] 10.222.0.102 () {54 vars in 926 bytes} [Wed Aug 24 15:39:32 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:39:32.577637 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a247fbd-db18-4be7-8f59-19344bd9fa98 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:32.578038 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a247fbd-db18-4be7-8f59-19344bd9fa98 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:32.578190 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a247fbd-db18-4be7-8f59-19344bd9fa98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:32.656589 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-1a247fbd-db18-4be7-8f59-19344bd9fa98 None None] MFA Rules not processed for user `0ee0002a76c3471f819b37fef510ef99`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:39:32.657802 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-1a247fbd-db18-4be7-8f59-19344bd9fa98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:32.770870 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 659/1355] 10.222.0.102 () {58 vars in 1023 bytes} [Wed Aug 24 15:39:32 2022] POST /identity/v3/auth/tokens => generated 3318 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:39:36.106699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fe6ada-3e4f-4899-8940-1e0f73efa4d3 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:36.107243 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fe6ada-3e4f-4899-8940-1e0f73efa4d3 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:36.107243 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fe6ada-3e4f-4899-8940-1e0f73efa4d3 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:36.108539 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 697/1356] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:39:36 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:39:36.113920 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6839a78-8c7a-4216-8c5f-339a55a7739b None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:36.114363 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6839a78-8c7a-4216-8c5f-339a55a7739b None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:36.114472 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6839a78-8c7a-4216-8c5f-339a55a7739b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:36.152382 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:36.198904 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-b6839a78-8c7a-4216-8c5f-339a55a7739b None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:39:36.200144 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-b6839a78-8c7a-4216-8c5f-339a55a7739b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:36.343417 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 660/1357] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:39:36 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:39:36.652635 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3049d0b0-4b7b-4d5e-9dbf-60bd882c3849 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:36.653265 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3049d0b0-4b7b-4d5e-9dbf-60bd882c3849 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:36.653722 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3049d0b0-4b7b-4d5e-9dbf-60bd882c3849 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:36.655264 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 698/1358] 10.222.0.102 () {54 vars in 967 bytes} [Wed Aug 24 15:39:36 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:39:36.668050 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa2f9fe-f999-4b70-9b78-28ca0c056cf1 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:36.668664 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa2f9fe-f999-4b70-9b78-28ca0c056cf1 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:36.669703 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa2f9fe-f999-4b70-9b78-28ca0c056cf1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:36.784629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-caa2f9fe-f999-4b70-9b78-28ca0c056cf1 None None] MFA Rules not processed for user `e0d544379fe44438ad10289b45124374`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:39:36.785593 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-caa2f9fe-f999-4b70-9b78-28ca0c056cf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:36.868863 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:36.964313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 661/1359] 10.222.0.102 () {58 vars in 1064 bytes} [Wed Aug 24 15:39:36 2022] POST /identity/v3/auth/tokens => generated 3382 bytes in 299 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:39:36.976881 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3049d0b0-4b7b-4d5e-9dbf-60bd882c3849 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:39:36.978387 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-3049d0b0-4b7b-4d5e-9dbf-60bd882c3849 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:37.144751 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3049d0b0-4b7b-4d5e-9dbf-60bd882c3849 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:39:37.225300 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:37.269565 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9fb57b-3a50-4d2f-a211-103aaad6d78a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:39:37.270172 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9fb57b-3a50-4d2f-a211-103aaad6d78a service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:37.270445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9fb57b-3a50-4d2f-a211-103aaad6d78a service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:37.270710 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9fb57b-3a50-4d2f-a211-103aaad6d78a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:37.273153 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-fd9fb57b-3a50-4d2f-a211-103aaad6d78a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:37.374025 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd9fb57b-3a50-4d2f-a211-103aaad6d78a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:39:37.375662 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd9fb57b-3a50-4d2f-a211-103aaad6d78a service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:39:37.394982 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 699/1360] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:39:36 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:39:39.564695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba471b7b-225e-4e44-8776-a620a01bee64 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:39.565629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba471b7b-225e-4e44-8776-a620a01bee64 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:39.566144 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba471b7b-225e-4e44-8776-a620a01bee64 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:39.567738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 662/1361] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:39:39 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:39:39.574634 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd45659-6d18-41b3-84f8-c3ea08671eae None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:39.575315 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd45659-6d18-41b3-84f8-c3ea08671eae None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:39.575315 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd45659-6d18-41b3-84f8-c3ea08671eae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:39.644951 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-bbd45659-6d18-41b3-84f8-c3ea08671eae None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:39:39.646153 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-bbd45659-6d18-41b3-84f8-c3ea08671eae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:39.688449 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:39.755924 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 700/1362] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:39:39 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:39:40.077385 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba471b7b-225e-4e44-8776-a620a01bee64 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:39:40.079950 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-ba471b7b-225e-4e44-8776-a620a01bee64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:40.250405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba471b7b-225e-4e44-8776-a620a01bee64 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:39:40.262035 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:40.404645 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad38c64-1512-43f9-9943-652813352efa service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:39:40.405471 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad38c64-1512-43f9-9943-652813352efa service nova] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:40.405936 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad38c64-1512-43f9-9943-652813352efa service nova] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:40.406405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad38c64-1512-43f9-9943-652813352efa service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:40.408222 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-aad38c64-1512-43f9-9943-652813352efa service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:40.520514 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad38c64-1512-43f9-9943-652813352efa service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:39:40.522892 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad38c64-1512-43f9-9943-652813352efa service nova] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:39:40.545608 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 663/1363] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:39:40 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 469 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:39:42.841407 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91192a5a-ec3a-48b6-add2-d59cf5b1d864 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:42.842468 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91192a5a-ec3a-48b6-add2-d59cf5b1d864 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:42.842977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91192a5a-ec3a-48b6-add2-d59cf5b1d864 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:42.846734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 701/1364] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:39:42 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:39:42.850564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef65dbc-93cf-452b-9d72-50c33f28c52b None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:42.851028 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef65dbc-93cf-452b-9d72-50c33f28c52b None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:42.851228 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef65dbc-93cf-452b-9d72-50c33f28c52b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:42.870112 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:42.937271 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-bef65dbc-93cf-452b-9d72-50c33f28c52b None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:39:42.938490 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-bef65dbc-93cf-452b-9d72-50c33f28c52b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:43.050846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 664/1365] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:39:42 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:39:43.356191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e043bb42-efec-4788-9981-def00b479e87 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:43.356811 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e043bb42-efec-4788-9981-def00b479e87 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:43.357088 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e043bb42-efec-4788-9981-def00b479e87 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:43.358852 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 702/1366] 10.222.0.102 () {54 vars in 967 bytes} [Wed Aug 24 15:39:43 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:39:43.364636 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2b6c8a-42ee-4230-82d9-8f73335c5170 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:43.365003 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2b6c8a-42ee-4230-82d9-8f73335c5170 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:43.365354 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2b6c8a-42ee-4230-82d9-8f73335c5170 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:43.464874 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-af2b6c8a-42ee-4230-82d9-8f73335c5170 None None] MFA Rules not processed for user `e0d544379fe44438ad10289b45124374`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:39:43.466129 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-af2b6c8a-42ee-4230-82d9-8f73335c5170 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:43.503954 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:43.597984 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 665/1367] 10.222.0.102 () {58 vars in 1064 bytes} [Wed Aug 24 15:39:43 2022] POST /identity/v3/auth/tokens => generated 3382 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:39:43.606697 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e043bb42-efec-4788-9981-def00b479e87 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:39:43.608252 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-e043bb42-efec-4788-9981-def00b479e87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:43.765275 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e043bb42-efec-4788-9981-def00b479e87 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:39:43.788715 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:43.905726 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f0467f-d799-4472-95f8-52b6d8b30d92 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:39:43.906418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f0467f-d799-4472-95f8-52b6d8b30d92 service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:43.906649 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f0467f-d799-4472-95f8-52b6d8b30d92 service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:43.906867 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f0467f-d799-4472-95f8-52b6d8b30d92 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:43.908868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-93f0467f-d799-4472-95f8-52b6d8b30d92 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:44.005428 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f0467f-d799-4472-95f8-52b6d8b30d92 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:39:44.007134 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f0467f-d799-4472-95f8-52b6d8b30d92 service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:39:44.027772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 703/1368] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:39:43 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 421 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:39:44.480167 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f527d6e6-a1a6-43f4-84af-6b985bfe1e35 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:44.480787 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f527d6e6-a1a6-43f4-84af-6b985bfe1e35 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:44.480787 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f527d6e6-a1a6-43f4-84af-6b985bfe1e35 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:44.482221 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 666/1369] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:39:44 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:39:44.487579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf95223-c124-4c60-98b3-135a2f81d5d0 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:44.488001 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf95223-c124-4c60-98b3-135a2f81d5d0 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:44.488376 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf95223-c124-4c60-98b3-135a2f81d5d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:44.523970 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:44.563352 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-bbf95223-c124-4c60-98b3-135a2f81d5d0 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:39:44.564753 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-bbf95223-c124-4c60-98b3-135a2f81d5d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:44.670561 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 704/1370] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:39:44 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:39:45.261771 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23987f0f-2fd3-45de-ab00-1b57221b7977 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:45.261771 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23987f0f-2fd3-45de-ab00-1b57221b7977 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:45.261771 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23987f0f-2fd3-45de-ab00-1b57221b7977 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:45.263249 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 667/1371] 10.222.0.102 () {54 vars in 969 bytes} [Wed Aug 24 15:39:45 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:39:45.268026 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfb7f2f-cb2a-42b7-8a5e-45e315f1ba2c None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:45.268513 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfb7f2f-cb2a-42b7-8a5e-45e315f1ba2c None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:45.268598 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfb7f2f-cb2a-42b7-8a5e-45e315f1ba2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:45.358066 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-cdfb7f2f-cb2a-42b7-8a5e-45e315f1ba2c None None] MFA Rules not processed for user `08fbd28d9593476b8753f8d64ee64f5b`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:39:45.359421 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-cdfb7f2f-cb2a-42b7-8a5e-45e315f1ba2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:45.409150 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:45.446004 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 705/1372] 10.222.0.102 () {58 vars in 1066 bytes} [Wed Aug 24 15:39:45 2022] POST /identity/v3/auth/tokens => generated 3130 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:39:45.453674 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23987f0f-2fd3-45de-ab00-1b57221b7977 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:39:45.455519 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-23987f0f-2fd3-45de-ab00-1b57221b7977 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:45.576679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23987f0f-2fd3-45de-ab00-1b57221b7977 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:39:45.648306 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:45.689688 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce833a13-c8a7-45ef-aca0-8677af2268c0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:39:45.690302 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce833a13-c8a7-45ef-aca0-8677af2268c0 service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:45.690625 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce833a13-c8a7-45ef-aca0-8677af2268c0 service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:45.690823 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce833a13-c8a7-45ef-aca0-8677af2268c0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:45.692435 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-ce833a13-c8a7-45ef-aca0-8677af2268c0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:45.816829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce833a13-c8a7-45ef-aca0-8677af2268c0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:39:45.818297 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce833a13-c8a7-45ef-aca0-8677af2268c0 service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:39:45.842792 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 668/1373] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:39:45 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:39:46.125382 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8681e36c-2c55-4fda-ac63-8ef447c5fec3 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:46.125724 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8681e36c-2c55-4fda-ac63-8ef447c5fec3 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:46.126008 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8681e36c-2c55-4fda-ac63-8ef447c5fec3 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:46.127501 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 706/1374] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:39:46 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:39:46.134029 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f427c5a3-0b67-4211-b2a6-baf962d33d31 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:46.134316 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f427c5a3-0b67-4211-b2a6-baf962d33d31 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:46.134773 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f427c5a3-0b67-4211-b2a6-baf962d33d31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:46.211915 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-f427c5a3-0b67-4211-b2a6-baf962d33d31 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:39:46.213234 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-f427c5a3-0b67-4211-b2a6-baf962d33d31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:46.234834 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:46.314739 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 669/1375] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:39:46 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:39:46.602723 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8681e36c-2c55-4fda-ac63-8ef447c5fec3 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:39:46.604483 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-8681e36c-2c55-4fda-ac63-8ef447c5fec3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:46.753923 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8681e36c-2c55-4fda-ac63-8ef447c5fec3 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:39:46.809589 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:46.876603 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c129db82-8f26-45d2-b917-73bb9e0de643 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:39:46.877216 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c129db82-8f26-45d2-b917-73bb9e0de643 service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:46.877511 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c129db82-8f26-45d2-b917-73bb9e0de643 service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:46.877771 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c129db82-8f26-45d2-b917-73bb9e0de643 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:46.879397 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-c129db82-8f26-45d2-b917-73bb9e0de643 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:46.973895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c129db82-8f26-45d2-b917-73bb9e0de643 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:39:46.976003 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c129db82-8f26-45d2-b917-73bb9e0de643 service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:39:46.994014 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 707/1376] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:39:46 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 392 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:39:48.950111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d278b9ef-1bb7-48a9-8439-02e114a107bf None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:48.950635 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d278b9ef-1bb7-48a9-8439-02e114a107bf None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:48.950635 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d278b9ef-1bb7-48a9-8439-02e114a107bf None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:48.952210 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 670/1377] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:39:48 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:39:48.957350 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9311b5b4-33ba-4a88-ba11-79dc0496b049 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:48.957806 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9311b5b4-33ba-4a88-ba11-79dc0496b049 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:48.957920 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9311b5b4-33ba-4a88-ba11-79dc0496b049 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:49.039446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-9311b5b4-33ba-4a88-ba11-79dc0496b049 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:39:49.040730 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-9311b5b4-33ba-4a88-ba11-79dc0496b049 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:49.047744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:49.142400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 708/1378] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:39:48 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:39:49.471133 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d278b9ef-1bb7-48a9-8439-02e114a107bf None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:39:49.472900 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-d278b9ef-1bb7-48a9-8439-02e114a107bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:49.648979 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d278b9ef-1bb7-48a9-8439-02e114a107bf None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:39:49.653384 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:49.778295 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81803092-a513-4101-b7da-9df626358888 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:39:49.778801 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81803092-a513-4101-b7da-9df626358888 service nova] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:49.779084 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81803092-a513-4101-b7da-9df626358888 service nova] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:49.779345 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81803092-a513-4101-b7da-9df626358888 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:49.780947 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-81803092-a513-4101-b7da-9df626358888 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:49.883249 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81803092-a513-4101-b7da-9df626358888 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:39:49.885089 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81803092-a513-4101-b7da-9df626358888 service nova] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:39:49.904882 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 671/1379] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:39:49 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 435 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:39:51.905684 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd98efc-a9cc-4212-be57-98a9106d1684 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:51.906216 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd98efc-a9cc-4212-be57-98a9106d1684 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:51.906216 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd98efc-a9cc-4212-be57-98a9106d1684 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:51.907844 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 709/1380] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:39:51 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:39:51.913531 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2597a0f8-d3dd-404e-bbfe-31c274ead9ee None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:51.914222 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2597a0f8-d3dd-404e-bbfe-31c274ead9ee None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:51.914222 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2597a0f8-d3dd-404e-bbfe-31c274ead9ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:51.926252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:51.985199 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-2597a0f8-d3dd-404e-bbfe-31c274ead9ee None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:39:51.986399 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-2597a0f8-d3dd-404e-bbfe-31c274ead9ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:52.093817 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 672/1381] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:39:51 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:39:52.389139 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd98efc-a9cc-4212-be57-98a9106d1684 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:39:52.390904 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-6bd98efc-a9cc-4212-be57-98a9106d1684 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:52.553962 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:52.568479 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd98efc-a9cc-4212-be57-98a9106d1684 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:39:52.705184 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96cd69b-4fa1-45d0-96f8-a92a26b963e1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:39:52.705818 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96cd69b-4fa1-45d0-96f8-a92a26b963e1 service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:52.706068 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96cd69b-4fa1-45d0-96f8-a92a26b963e1 service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:52.706337 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96cd69b-4fa1-45d0-96f8-a92a26b963e1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:52.707946 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-b96cd69b-4fa1-45d0-96f8-a92a26b963e1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:52.804362 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96cd69b-4fa1-45d0-96f8-a92a26b963e1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:39:52.805973 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96cd69b-4fa1-45d0-96f8-a92a26b963e1 service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:39:52.817354 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:52.831442 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 710/1382] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:39:52 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 443 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:39:55.171612 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27e5557-2a14-4aba-ad7b-01f75ce7fe8e None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:55.172323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27e5557-2a14-4aba-ad7b-01f75ce7fe8e None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:55.172323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27e5557-2a14-4aba-ad7b-01f75ce7fe8e None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:55.173693 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 673/1383] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:39:55 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:39:55.180164 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ac22a8-0f6e-48c1-983c-9d20b3ab019d None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:55.180623 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ac22a8-0f6e-48c1-983c-9d20b3ab019d None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:55.180868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ac22a8-0f6e-48c1-983c-9d20b3ab019d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:55.266364 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-d6ac22a8-0f6e-48c1-983c-9d20b3ab019d None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:39:55.267994 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-d6ac22a8-0f6e-48c1-983c-9d20b3ab019d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:55.378365 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 711/1384] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:39:55 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:39:55.692151 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27e5557-2a14-4aba-ad7b-01f75ce7fe8e None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:39:55.695027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-a27e5557-2a14-4aba-ad7b-01f75ce7fe8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:55.796905 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:55.886261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27e5557-2a14-4aba-ad7b-01f75ce7fe8e None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:39:56.018384 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2652eba-1a82-43f1-b951-975ce07df636 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:39:56.018998 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2652eba-1a82-43f1-b951-975ce07df636 service nova] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:56.019297 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2652eba-1a82-43f1-b951-975ce07df636 service nova] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:56.019586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2652eba-1a82-43f1-b951-975ce07df636 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:56.021286 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-d2652eba-1a82-43f1-b951-975ce07df636 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:56.075085 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:56.118086 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2652eba-1a82-43f1-b951-975ce07df636 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:39:56.119801 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2652eba-1a82-43f1-b951-975ce07df636 service nova] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:39:56.138784 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 674/1385] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:39:55 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 447 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:39:58.102511 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390eaec3-a6b4-4a17-9a7a-bc25506cbbc9 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:58.103041 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390eaec3-a6b4-4a17-9a7a-bc25506cbbc9 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:58.103041 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390eaec3-a6b4-4a17-9a7a-bc25506cbbc9 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:58.104367 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 712/1386] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:39:58 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:39:58.110236 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0dc842-f3c5-4789-b286-e60319c68091 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:58.110789 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0dc842-f3c5-4789-b286-e60319c68091 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:58.110789 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0dc842-f3c5-4789-b286-e60319c68091 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:58.183554 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-2c0dc842-f3c5-4789-b286-e60319c68091 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:39:58.184799 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-2c0dc842-f3c5-4789-b286-e60319c68091 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:58.288497 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 675/1387] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:39:58 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:39:58.591933 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390eaec3-a6b4-4a17-9a7a-bc25506cbbc9 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:39:58.593533 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-390eaec3-a6b4-4a17-9a7a-bc25506cbbc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:58.678506 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:58.752316 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390eaec3-a6b4-4a17-9a7a-bc25506cbbc9 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:39:58.878574 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c9ef743-8314-46d1-9839-299bf907f1d9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:39:58.879196 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9ef743-8314-46d1-9839-299bf907f1d9 service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:39:58.879499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9ef743-8314-46d1-9839-299bf907f1d9 service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:39:58.879809 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9ef743-8314-46d1-9839-299bf907f1d9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:39:58.882177 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-0c9ef743-8314-46d1-9839-299bf907f1d9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:39:58.923940 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:39:58.989510 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c9ef743-8314-46d1-9839-299bf907f1d9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:39:58.992193 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c9ef743-8314-46d1-9839-299bf907f1d9 service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:39:59.015208 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 713/1388] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:39:58 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 424 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 15:40:01.526140 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0a7761-0b39-4c22-9c0c-fd2379a29539 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:01.526713 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0a7761-0b39-4c22-9c0c-fd2379a29539 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:01.526713 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0a7761-0b39-4c22-9c0c-fd2379a29539 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:01.528041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 676/1389] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:40:01 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:40:01.533394 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25ff147-c70e-4f01-849b-7c50cb22c983 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:01.533811 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25ff147-c70e-4f01-849b-7c50cb22c983 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:01.533950 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25ff147-c70e-4f01-849b-7c50cb22c983 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:01.603157 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-b25ff147-c70e-4f01-849b-7c50cb22c983 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:40:01.604355 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-b25ff147-c70e-4f01-849b-7c50cb22c983 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:01.665531 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64987d83-f05c-4007-a38f-1e93f7b4cb72 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:01.665821 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64987d83-f05c-4007-a38f-1e93f7b4cb72 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:01.666105 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64987d83-f05c-4007-a38f-1e93f7b4cb72 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:01.667449 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 677/1390] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:40:01 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:40:01.672416 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1cc000-1b9c-4210-a670-ca33402794f6 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:01.672704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1cc000-1b9c-4210-a670-ca33402794f6 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:01.672980 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1cc000-1b9c-4210-a670-ca33402794f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:01.712043 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:01.725541 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 714/1391] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:40:01 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:40:01.746689 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-4c1cc000-1b9c-4210-a670-ca33402794f6 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:40:01.747964 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-4c1cc000-1b9c-4210-a670-ca33402794f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:01.854044 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 678/1392] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:40:01 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:40:02.023703 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25ff147-c70e-4f01-849b-7c50cb22c983 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:40:02.025461 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-b25ff147-c70e-4f01-849b-7c50cb22c983 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:02.047064 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:02.154004 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25ff147-c70e-4f01-849b-7c50cb22c983 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:40:02.173346 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce592c1-87c4-4fa3-898c-53008aa4a890 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:02.173638 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce592c1-87c4-4fa3-898c-53008aa4a890 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:02.173916 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce592c1-87c4-4fa3-898c-53008aa4a890 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:02.175693 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 679/1393] 10.222.0.102 () {54 vars in 972 bytes} [Wed Aug 24 15:40:02 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:40:02.180367 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e73edcd-31c1-4e41-9a25-d29591fe6959 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:02.180654 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e73edcd-31c1-4e41-9a25-d29591fe6959 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:02.181025 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e73edcd-31c1-4e41-9a25-d29591fe6959 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:02.259421 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-1e73edcd-31c1-4e41-9a25-d29591fe6959 None None] MFA Rules not processed for user `8c6f06186c1a4626997db5a40c31cbb9`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:40:02.260659 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-1e73edcd-31c1-4e41-9a25-d29591fe6959 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:02.278775 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063e5920-4c35-4b06-8ace-ebeb3bb3f350 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:40:02.279390 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063e5920-4c35-4b06-8ace-ebeb3bb3f350 service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:02.279729 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063e5920-4c35-4b06-8ace-ebeb3bb3f350 service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:02.279998 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063e5920-4c35-4b06-8ace-ebeb3bb3f350 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:02.281687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-063e5920-4c35-4b06-8ace-ebeb3bb3f350 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:02.291201 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:02.312746 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:02.347445 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 680/1394] 10.222.0.102 () {58 vars in 1069 bytes} [Wed Aug 24 15:40:02 2022] POST /identity/v3/auth/tokens => generated 3130 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:40:02.355896 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e73edcd-31c1-4e41-9a25-d29591fe6959 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:40:02.363742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e73edcd-31c1-4e41-9a25-d29591fe6959 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:40:02.382375 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063e5920-4c35-4b06-8ace-ebeb3bb3f350 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:40:02.384163 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063e5920-4c35-4b06-8ace-ebeb3bb3f350 service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:40:02.403492 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 715/1395] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:40:02 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:40:02.445552 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a75234-ed82-43a4-8e95-5718e42c3c8a service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c6f06186c1a4626997db5a40c31cbb9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:40:02.446163 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a75234-ed82-43a4-8e95-5718e42c3c8a service glance] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:02.446449 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a75234-ed82-43a4-8e95-5718e42c3c8a service glance] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:02.446723 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a75234-ed82-43a4-8e95-5718e42c3c8a service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:02.448481 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-98a75234-ed82-43a4-8e95-5718e42c3c8a service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:02.576584 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a75234-ed82-43a4-8e95-5718e42c3c8a service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:40:02.578777 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a75234-ed82-43a4-8e95-5718e42c3c8a service glance] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:40:02.595169 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:02.600926 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 681/1396] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:40:02 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 245 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 15:40:03.346927 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9261f495-f6f0-4110-a9d6-ce2a328ba9e6 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:03.347373 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9261f495-f6f0-4110-a9d6-ce2a328ba9e6 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:03.347488 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9261f495-f6f0-4110-a9d6-ce2a328ba9e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:03.417643 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-9261f495-f6f0-4110-a9d6-ce2a328ba9e6 None None] MFA Rules not processed for user `a6c17620ead14cb49d962990e1b1e0d8`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:40:03.418853 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-9261f495-f6f0-4110-a9d6-ce2a328ba9e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:03.504921 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:03.508879 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 716/1397] 10.222.0.102 () {58 vars in 1014 bytes} [Wed Aug 24 15:40:03 2022] POST /identity/v3/auth/tokens => generated 3205 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:40:03.554216 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4a5bf7-ccff-4a61-a743-d17862ae3076 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:03.554502 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4a5bf7-ccff-4a61-a743-d17862ae3076 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:03.554775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4a5bf7-ccff-4a61-a743-d17862ae3076 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:03.556173 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 682/1398] 10.222.0.102 () {54 vars in 948 bytes} [Wed Aug 24 15:40:03 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:40:03.562540 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a286d3-05e4-444f-9904-20d33351f7c8 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:03.562830 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a286d3-05e4-444f-9904-20d33351f7c8 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:03.563108 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a286d3-05e4-444f-9904-20d33351f7c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:03.643818 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-98a286d3-05e4-444f-9904-20d33351f7c8 None None] MFA Rules not processed for user `6a070d4fd8c4466c857bd03965c11349`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:40:03.645038 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-98a286d3-05e4-444f-9904-20d33351f7c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:03.754537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 717/1399] 10.222.0.102 () {58 vars in 1045 bytes} [Wed Aug 24 15:40:03 2022] POST /identity/v3/auth/tokens => generated 3314 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:40:03.763881 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4a5bf7-ccff-4a61-a743-d17862ae3076 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:40:03.765427 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-1a4a5bf7-ccff-4a61-a743-d17862ae3076 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:03.910923 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4a5bf7-ccff-4a61-a743-d17862ae3076 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:40:03.997400 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:04.019012 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15684659-79c0-4a76-bbb7-02c855145dd4 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a070d4fd8c4466c857bd03965c11349', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:40:04.019618 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15684659-79c0-4a76-bbb7-02c855145dd4 service swift] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:04.019924 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15684659-79c0-4a76-bbb7-02c855145dd4 service swift] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:04.020195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15684659-79c0-4a76-bbb7-02c855145dd4 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:04.022187 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-15684659-79c0-4a76-bbb7-02c855145dd4 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:04.111365 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15684659-79c0-4a76-bbb7-02c855145dd4 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:40:04.113055 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15684659-79c0-4a76-bbb7-02c855145dd4 service swift] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:40:04.122649 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 683/1400] 10.222.0.102 () {60 vars in 1389 bytes} [Wed Aug 24 15:40:03 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 24 15:40:04.378452 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bd3185-3232-45cf-be36-82e6b97b5ee0 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:04.379005 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bd3185-3232-45cf-be36-82e6b97b5ee0 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:04.379005 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bd3185-3232-45cf-be36-82e6b97b5ee0 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:04.380555 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 718/1401] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:40:04 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:40:04.388229 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34661a35-6e76-4242-9320-b4e848731262 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:04.388756 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34661a35-6e76-4242-9320-b4e848731262 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:04.389204 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34661a35-6e76-4242-9320-b4e848731262 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:04.466274 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-34661a35-6e76-4242-9320-b4e848731262 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:40:04.467564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-34661a35-6e76-4242-9320-b4e848731262 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:04.545738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:04.579910 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 684/1402] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:40:04 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:40:04.873344 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61bd3185-3232-45cf-be36-82e6b97b5ee0 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:40:04.873344 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-61bd3185-3232-45cf-be36-82e6b97b5ee0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:04.950212 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:05.104918 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61bd3185-3232-45cf-be36-82e6b97b5ee0 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:40:05.259294 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85b1270-a89f-4e57-9a02-0cb9949928f8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:40:05.259950 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85b1270-a89f-4e57-9a02-0cb9949928f8 service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:05.260237 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85b1270-a89f-4e57-9a02-0cb9949928f8 service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:05.260515 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85b1270-a89f-4e57-9a02-0cb9949928f8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:05.264018 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-b85b1270-a89f-4e57-9a02-0cb9949928f8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:05.294156 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:05.398779 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b85b1270-a89f-4e57-9a02-0cb9949928f8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:40:05.400444 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b85b1270-a89f-4e57-9a02-0cb9949928f8 service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:40:05.421070 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 719/1403] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:40:04 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 550 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:40:07.472823 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce897653-e621-4e3c-9b8a-8051752957c7 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:07.473397 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce897653-e621-4e3c-9b8a-8051752957c7 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:07.473397 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce897653-e621-4e3c-9b8a-8051752957c7 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:07.474961 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 685/1404] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:40:07 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:40:07.480561 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba140184-9206-437b-b75d-8e5b4254af44 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:07.481027 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba140184-9206-437b-b75d-8e5b4254af44 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:07.481141 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba140184-9206-437b-b75d-8e5b4254af44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:07.563213 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-ba140184-9206-437b-b75d-8e5b4254af44 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:40:07.564466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-ba140184-9206-437b-b75d-8e5b4254af44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:07.668808 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:07.676299 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 720/1405] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:40:07 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:40:07.968658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce897653-e621-4e3c-9b8a-8051752957c7 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:40:07.970224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-ce897653-e621-4e3c-9b8a-8051752957c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:08.157100 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce897653-e621-4e3c-9b8a-8051752957c7 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:40:08.222373 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:08.290862 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e58219-1698-46e0-8c31-e1ff20dd085b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:40:08.291472 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e58219-1698-46e0-8c31-e1ff20dd085b service nova] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:08.291782 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e58219-1698-46e0-8c31-e1ff20dd085b service nova] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:08.292050 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e58219-1698-46e0-8c31-e1ff20dd085b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:08.293650 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-85e58219-1698-46e0-8c31-e1ff20dd085b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:08.393881 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e58219-1698-46e0-8c31-e1ff20dd085b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:40:08.395543 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e58219-1698-46e0-8c31-e1ff20dd085b service nova] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:40:08.417921 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 686/1406] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:40:07 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 450 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 15:40:10.477492 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3b45a9-57f2-4847-b54e-a616209de20e None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:10.478052 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3b45a9-57f2-4847-b54e-a616209de20e None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:10.478052 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3b45a9-57f2-4847-b54e-a616209de20e None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:10.480007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 721/1407] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:40:10 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:40:10.484995 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9047336-e2ae-4eb6-80ac-12d484b4bb71 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:10.485449 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9047336-e2ae-4eb6-80ac-12d484b4bb71 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:10.485563 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9047336-e2ae-4eb6-80ac-12d484b4bb71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:10.557202 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-d9047336-e2ae-4eb6-80ac-12d484b4bb71 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:40:10.558510 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-d9047336-e2ae-4eb6-80ac-12d484b4bb71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:10.573402 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:10.673874 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 687/1408] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:40:10 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:40:10.966612 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a3b45a9-57f2-4847-b54e-a616209de20e None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:40:10.968193 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-3a3b45a9-57f2-4847-b54e-a616209de20e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:11.138431 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a3b45a9-57f2-4847-b54e-a616209de20e None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:40:11.217563 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:11.270123 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28730ff9-5f9d-4cfa-b0a0-969964732c5c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:40:11.270959 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28730ff9-5f9d-4cfa-b0a0-969964732c5c service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:11.271302 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28730ff9-5f9d-4cfa-b0a0-969964732c5c service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:11.271693 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28730ff9-5f9d-4cfa-b0a0-969964732c5c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:11.273565 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-28730ff9-5f9d-4cfa-b0a0-969964732c5c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:11.380579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28730ff9-5f9d-4cfa-b0a0-969964732c5c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:40:11.382969 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28730ff9-5f9d-4cfa-b0a0-969964732c5c service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:40:11.405084 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 722/1409] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:40:10 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 439 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:40:20.671580 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=38 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:21.401029 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=64 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:26.765555 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd9461c-cbab-4ae8-9353-f261d0a1e493 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:26.766133 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd9461c-cbab-4ae8-9353-f261d0a1e493 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:26.766133 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd9461c-cbab-4ae8-9353-f261d0a1e493 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:26.767509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 688/1410] 10.222.0.102 () {54 vars in 923 bytes} [Wed Aug 24 15:40:26 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:40:26.773492 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66e55a9-1edc-48ea-946b-9df9f84dc095 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:26.773924 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66e55a9-1edc-48ea-946b-9df9f84dc095 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:26.774056 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66e55a9-1edc-48ea-946b-9df9f84dc095 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:26.854989 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-d66e55a9-1edc-48ea-946b-9df9f84dc095 None None] MFA Rules not processed for user `0ee0002a76c3471f819b37fef510ef99`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:40:26.856257 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-d66e55a9-1edc-48ea-946b-9df9f84dc095 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:26.970690 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 723/1411] 10.222.0.102 () {58 vars in 1020 bytes} [Wed Aug 24 15:40:26 2022] POST /identity/v3/auth/tokens => generated 3318 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:40:26.986863 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cdf4a6-9735-4244-82cf-eb2dd73df5fc None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:26.987148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cdf4a6-9735-4244-82cf-eb2dd73df5fc None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:26.987422 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cdf4a6-9735-4244-82cf-eb2dd73df5fc None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:26.988799 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 689/1412] 10.222.0.102 () {54 vars in 923 bytes} [Wed Aug 24 15:40:26 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:40:26.994874 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37889a90-9b28-4d30-8998-29f8f964014f None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:26.995167 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37889a90-9b28-4d30-8998-29f8f964014f None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:26.995451 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37889a90-9b28-4d30-8998-29f8f964014f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:27.074787 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:27.078142 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-37889a90-9b28-4d30-8998-29f8f964014f None None] MFA Rules not processed for user `e0d544379fe44438ad10289b45124374`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:40:27.079567 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-37889a90-9b28-4d30-8998-29f8f964014f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:27.196906 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 724/1413] 10.222.0.102 () {58 vars in 1020 bytes} [Wed Aug 24 15:40:26 2022] POST /identity/v3/auth/tokens => generated 3382 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:40:29.597159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13a7bc5-3ad5-4dbd-b82d-d70d252226d5 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:29.597681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13a7bc5-3ad5-4dbd-b82d-d70d252226d5 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:29.597681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13a7bc5-3ad5-4dbd-b82d-d70d252226d5 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:29.598971 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 690/1414] 127.0.0.1 () {54 vars in 917 bytes} [Wed Aug 24 15:40:29 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:40:29.604220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fd44e0-b3eb-43e0-a312-617d6928fa24 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:29.604638 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fd44e0-b3eb-43e0-a312-617d6928fa24 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:29.604760 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fd44e0-b3eb-43e0-a312-617d6928fa24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:29.670287 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-c6fd44e0-b3eb-43e0-a312-617d6928fa24 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:40:29.671409 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-c6fd44e0-b3eb-43e0-a312-617d6928fa24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:29.726655 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:29.780369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 725/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Aug 24 15:40:29 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:40:30.086427 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13a7bc5-3ad5-4dbd-b82d-d70d252226d5 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:40:30.088143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-d13a7bc5-3ad5-4dbd-b82d-d70d252226d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:30.354929 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13a7bc5-3ad5-4dbd-b82d-d70d252226d5 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:40:30.432834 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:30.473029 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fef8f1a-87c0-40d6-9109-8114d494d292 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:40:30.473657 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fef8f1a-87c0-40d6-9109-8114d494d292 service nova] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:30.473934 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fef8f1a-87c0-40d6-9109-8114d494d292 service nova] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:30.474201 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fef8f1a-87c0-40d6-9109-8114d494d292 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:30.476640 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-3fef8f1a-87c0-40d6-9109-8114d494d292 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:30.567241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fef8f1a-87c0-40d6-9109-8114d494d292 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:40:30.568921 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fef8f1a-87c0-40d6-9109-8114d494d292 service nova] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:40:30.586992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 691/1416] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:40:30 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 501 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:40:31.505149 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208b193c-17c0-43bb-a79f-166f02b6c071 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:31.505728 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208b193c-17c0-43bb-a79f-166f02b6c071 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:31.505728 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208b193c-17c0-43bb-a79f-166f02b6c071 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:31.507231 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 726/1417] 127.0.0.1 () {54 vars in 901 bytes} [Wed Aug 24 15:40:31 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:40:31.512318 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93afa9de-925b-4fb9-9d4c-be548eb3fe96 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:31.512736 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93afa9de-925b-4fb9-9d4c-be548eb3fe96 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:31.512853 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93afa9de-925b-4fb9-9d4c-be548eb3fe96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:31.588887 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-93afa9de-925b-4fb9-9d4c-be548eb3fe96 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:40:31.590085 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-93afa9de-925b-4fb9-9d4c-be548eb3fe96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:31.617429 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:31.689448 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 692/1418] 127.0.0.1 () {58 vars in 998 bytes} [Wed Aug 24 15:40:31 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:40:31.845946 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208b193c-17c0-43bb-a79f-166f02b6c071 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:40:31.847599 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-208b193c-17c0-43bb-a79f-166f02b6c071 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:31.997758 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208b193c-17c0-43bb-a79f-166f02b6c071 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:40:32.045149 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:32.123257 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d98f60f-505a-4402-bd31-5fddf45a597f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:40:32.123895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d98f60f-505a-4402-bd31-5fddf45a597f service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:32.124179 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d98f60f-505a-4402-bd31-5fddf45a597f service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:32.124453 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d98f60f-505a-4402-bd31-5fddf45a597f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:32.126072 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-2d98f60f-505a-4402-bd31-5fddf45a597f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:32.220615 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d98f60f-505a-4402-bd31-5fddf45a597f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:40:32.222247 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d98f60f-505a-4402-bd31-5fddf45a597f service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:40:32.240966 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 727/1419] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:40:31 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 395 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:40:33.133391 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31acc64b-606a-4ec2-ae69-ead062032f0d None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:33.133918 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31acc64b-606a-4ec2-ae69-ead062032f0d None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:33.133918 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31acc64b-606a-4ec2-ae69-ead062032f0d None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:33.135255 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 693/1420] 127.0.0.1 () {54 vars in 901 bytes} [Wed Aug 24 15:40:33 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:40:33.140347 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b7ce6c-7abf-4a2d-8c87-a7b3243b342b None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:33.140796 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b7ce6c-7abf-4a2d-8c87-a7b3243b342b None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:33.140936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b7ce6c-7abf-4a2d-8c87-a7b3243b342b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:33.196118 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:33.218275 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-88b7ce6c-7abf-4a2d-8c87-a7b3243b342b None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:40:33.219474 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-88b7ce6c-7abf-4a2d-8c87-a7b3243b342b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:33.314670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 728/1421] 127.0.0.1 () {58 vars in 998 bytes} [Wed Aug 24 15:40:33 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:40:33.467937 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31acc64b-606a-4ec2-ae69-ead062032f0d None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:40:33.469257 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-31acc64b-606a-4ec2-ae69-ead062032f0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:33.614173 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31acc64b-606a-4ec2-ae69-ead062032f0d None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:40:33.627232 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:33.729965 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50e462f-c3d9-4dc8-8d22-9524cecf2760 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:40:33.730428 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50e462f-c3d9-4dc8-8d22-9524cecf2760 service nova] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:33.730709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50e462f-c3d9-4dc8-8d22-9524cecf2760 service nova] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:33.730981 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50e462f-c3d9-4dc8-8d22-9524cecf2760 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:33.732619 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-d50e462f-c3d9-4dc8-8d22-9524cecf2760 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:33.822148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50e462f-c3d9-4dc8-8d22-9524cecf2760 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:40:33.823790 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50e462f-c3d9-4dc8-8d22-9524cecf2760 service nova] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:40:33.842614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 694/1422] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:40:33 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:40:34.736143 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a2c169-0254-468c-b8b9-45ddbc9ee6d7 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:34.736676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a2c169-0254-468c-b8b9-45ddbc9ee6d7 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:34.736676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a2c169-0254-468c-b8b9-45ddbc9ee6d7 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:34.737972 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 729/1423] 127.0.0.1 () {54 vars in 901 bytes} [Wed Aug 24 15:40:34 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:40:34.743575 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7102c566-5f21-4f64-ad8f-c9a75bff4e3a None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:34.744021 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7102c566-5f21-4f64-ad8f-c9a75bff4e3a None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:34.744137 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7102c566-5f21-4f64-ad8f-c9a75bff4e3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:34.764132 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:34.822486 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-7102c566-5f21-4f64-ad8f-c9a75bff4e3a None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:40:34.823746 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-7102c566-5f21-4f64-ad8f-c9a75bff4e3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:34.925303 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 695/1424] 127.0.0.1 () {58 vars in 998 bytes} [Wed Aug 24 15:40:34 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:40:35.079667 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a2c169-0254-468c-b8b9-45ddbc9ee6d7 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:40:35.081122 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-f1a2c169-0254-468c-b8b9-45ddbc9ee6d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:35.195140 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:35.230472 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a2c169-0254-468c-b8b9-45ddbc9ee6d7 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:40:35.344489 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2726d567-ab6c-453a-9ca6-883b3327c24e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:40:35.345014 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2726d567-ab6c-453a-9ca6-883b3327c24e service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:35.345299 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2726d567-ab6c-453a-9ca6-883b3327c24e service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:35.345574 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2726d567-ab6c-453a-9ca6-883b3327c24e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:35.347193 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-2726d567-ab6c-453a-9ca6-883b3327c24e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:35.425504 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:35.440878 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2726d567-ab6c-453a-9ca6-883b3327c24e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:40:35.442511 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2726d567-ab6c-453a-9ca6-883b3327c24e service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:40:35.461949 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 730/1425] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:40:35 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:40:36.339146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7873a19e-0c6f-4786-9246-3781e586b114 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:36.339732 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7873a19e-0c6f-4786-9246-3781e586b114 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:36.339732 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7873a19e-0c6f-4786-9246-3781e586b114 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:36.341095 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 696/1426] 127.0.0.1 () {54 vars in 901 bytes} [Wed Aug 24 15:40:36 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:40:36.346042 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd261a2-2224-464f-9898-eeee2544084f None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:36.346457 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd261a2-2224-464f-9898-eeee2544084f None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:36.346590 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd261a2-2224-464f-9898-eeee2544084f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:36.424446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-5fd261a2-2224-464f-9898-eeee2544084f None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:40:36.425677 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-5fd261a2-2224-464f-9898-eeee2544084f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:36.526469 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 731/1427] 127.0.0.1 () {58 vars in 998 bytes} [Wed Aug 24 15:40:36 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:40:36.683831 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7873a19e-0c6f-4786-9246-3781e586b114 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:40:36.685482 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-7873a19e-0c6f-4786-9246-3781e586b114 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:36.774575 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:36.841940 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7873a19e-0c6f-4786-9246-3781e586b114 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:40:36.961324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcfb7c6-85c1-44d7-a409-8a113726ead0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:40:36.961848 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcfb7c6-85c1-44d7-a409-8a113726ead0 service nova] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:36.962130 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcfb7c6-85c1-44d7-a409-8a113726ead0 service nova] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:36.962404 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcfb7c6-85c1-44d7-a409-8a113726ead0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:36.964059 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-7fcfb7c6-85c1-44d7-a409-8a113726ead0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:37.011769 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:37.056354 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fcfb7c6-85c1-44d7-a409-8a113726ead0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:40:37.057953 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fcfb7c6-85c1-44d7-a409-8a113726ead0 service nova] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:40:37.076971 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 697/1428] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:40:36 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:40:37.927354 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bceecd4-57f7-4afb-85de-dd496d102e32 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:37.928563 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bceecd4-57f7-4afb-85de-dd496d102e32 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:37.928563 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bceecd4-57f7-4afb-85de-dd496d102e32 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:37.930096 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 732/1429] 127.0.0.1 () {54 vars in 901 bytes} [Wed Aug 24 15:40:37 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:40:37.935215 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5903dbf4-1765-488b-94c4-131d39d3a224 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:37.935684 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5903dbf4-1765-488b-94c4-131d39d3a224 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:37.935799 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5903dbf4-1765-488b-94c4-131d39d3a224 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:38.011792 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-5903dbf4-1765-488b-94c4-131d39d3a224 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:40:38.013003 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-5903dbf4-1765-488b-94c4-131d39d3a224 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:38.124323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 698/1430] 127.0.0.1 () {58 vars in 998 bytes} [Wed Aug 24 15:40:37 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:40:38.282372 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bceecd4-57f7-4afb-85de-dd496d102e32 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:40:38.283971 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-1bceecd4-57f7-4afb-85de-dd496d102e32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:38.338789 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:38.443768 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bceecd4-57f7-4afb-85de-dd496d102e32 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:40:38.558379 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d972484-1bc0-4d0a-a5f3-ba518025f148 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:40:38.558985 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d972484-1bc0-4d0a-a5f3-ba518025f148 service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:38.559263 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d972484-1bc0-4d0a-a5f3-ba518025f148 service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:38.559539 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d972484-1bc0-4d0a-a5f3-ba518025f148 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:38.561179 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-3d972484-1bc0-4d0a-a5f3-ba518025f148 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:38.568029 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:38.653730 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d972484-1bc0-4d0a-a5f3-ba518025f148 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:40:38.655344 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d972484-1bc0-4d0a-a5f3-ba518025f148 service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:40:38.674511 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 733/1431] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:40:38 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:40:39.509296 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad23b873-1f05-4a39-a5d2-9cb8028fb3b6 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:39.509900 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad23b873-1f05-4a39-a5d2-9cb8028fb3b6 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:39.509900 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad23b873-1f05-4a39-a5d2-9cb8028fb3b6 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:39.511196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 699/1432] 127.0.0.1 () {54 vars in 901 bytes} [Wed Aug 24 15:40:39 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:40:39.516220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394fa1fb-70b2-4f82-82fe-75d07cc0e18d None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:39.516655 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394fa1fb-70b2-4f82-82fe-75d07cc0e18d None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:39.516815 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394fa1fb-70b2-4f82-82fe-75d07cc0e18d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:39.598264 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-394fa1fb-70b2-4f82-82fe-75d07cc0e18d None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:40:39.599543 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-394fa1fb-70b2-4f82-82fe-75d07cc0e18d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:39.669587 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:39.698845 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 734/1433] 127.0.0.1 () {58 vars in 998 bytes} [Wed Aug 24 15:40:39 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:40:39.854554 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad23b873-1f05-4a39-a5d2-9cb8028fb3b6 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:40:39.856151 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-ad23b873-1f05-4a39-a5d2-9cb8028fb3b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:39.873597 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:40.003991 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad23b873-1f05-4a39-a5d2-9cb8028fb3b6 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:40:40.103023 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:40.129090 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9247a32-c565-48eb-b52b-d83ef63ab12d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:40:40.129735 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9247a32-c565-48eb-b52b-d83ef63ab12d service nova] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:40.129972 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9247a32-c565-48eb-b52b-d83ef63ab12d service nova] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:40.130239 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9247a32-c565-48eb-b52b-d83ef63ab12d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:40.132496 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-a9247a32-c565-48eb-b52b-d83ef63ab12d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:40.221174 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9247a32-c565-48eb-b52b-d83ef63ab12d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:40:40.222764 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9247a32-c565-48eb-b52b-d83ef63ab12d service nova] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:40:40.240575 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 700/1434] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:40:39 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:40:41.092093 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c585a0-f23e-4567-bb2c-95068852c13f None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:41.092670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c585a0-f23e-4567-bb2c-95068852c13f None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:41.092670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c585a0-f23e-4567-bb2c-95068852c13f None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:41.093995 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 735/1435] 127.0.0.1 () {54 vars in 901 bytes} [Wed Aug 24 15:40:41 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:40:41.099106 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6f160d-e8fa-4c4b-bf5e-0b2f06deb68c None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:41.099588 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6f160d-e8fa-4c4b-bf5e-0b2f06deb68c None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:41.099588 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6f160d-e8fa-4c4b-bf5e-0b2f06deb68c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:41.176679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-6b6f160d-e8fa-4c4b-bf5e-0b2f06deb68c None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:40:41.178189 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-6b6f160d-e8fa-4c4b-bf5e-0b2f06deb68c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:41.220169 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:41.276763 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 701/1436] 127.0.0.1 () {58 vars in 998 bytes} [Wed Aug 24 15:40:41 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:40:41.429688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c585a0-f23e-4567-bb2c-95068852c13f None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:40:41.431213 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-d7c585a0-f23e-4567-bb2c-95068852c13f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:41.583470 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c585a0-f23e-4567-bb2c-95068852c13f None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:40:41.639170 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:41.700041 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1135509-b3af-427a-b16c-22fb8a2a0890 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:40:41.700656 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1135509-b3af-427a-b16c-22fb8a2a0890 service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:41.700940 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1135509-b3af-427a-b16c-22fb8a2a0890 service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:41.701212 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1135509-b3af-427a-b16c-22fb8a2a0890 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:41.702832 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-a1135509-b3af-427a-b16c-22fb8a2a0890 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:41.793554 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1135509-b3af-427a-b16c-22fb8a2a0890 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:40:41.795188 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1135509-b3af-427a-b16c-22fb8a2a0890 service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:40:41.813346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 736/1437] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:40:41 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:40:42.621046 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7641a963-a925-4d59-8a27-2a757a9114f9 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:42.621567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7641a963-a925-4d59-8a27-2a757a9114f9 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:42.621567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7641a963-a925-4d59-8a27-2a757a9114f9 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:42.622850 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 702/1438] 127.0.0.1 () {54 vars in 901 bytes} [Wed Aug 24 15:40:42 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:40:42.627690 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f97b03-8df6-49e0-bde0-c06712dff64d None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:42.628073 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f97b03-8df6-49e0-bde0-c06712dff64d None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:42.628242 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f97b03-8df6-49e0-bde0-c06712dff64d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:42.727249 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-54f97b03-8df6-49e0-bde0-c06712dff64d None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:40:42.728543 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-54f97b03-8df6-49e0-bde0-c06712dff64d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:42.731608 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:42.832796 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 737/1439] 127.0.0.1 () {58 vars in 998 bytes} [Wed Aug 24 15:40:42 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:40:42.987368 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7641a963-a925-4d59-8a27-2a757a9114f9 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:40:42.988949 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-7641a963-a925-4d59-8a27-2a757a9114f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:43.135734 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7641a963-a925-4d59-8a27-2a757a9114f9 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:40:43.162958 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:43.248676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3345b9d4-131d-44da-a4c2-4b6c9a8bba2e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:40:43.249223 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3345b9d4-131d-44da-a4c2-4b6c9a8bba2e service nova] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:43.249503 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3345b9d4-131d-44da-a4c2-4b6c9a8bba2e service nova] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:43.249773 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3345b9d4-131d-44da-a4c2-4b6c9a8bba2e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:43.251375 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-3345b9d4-131d-44da-a4c2-4b6c9a8bba2e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:43.340861 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3345b9d4-131d-44da-a4c2-4b6c9a8bba2e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:40:43.342454 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3345b9d4-131d-44da-a4c2-4b6c9a8bba2e service nova] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:40:43.360645 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 703/1440] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:40:42 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:40:45.188082 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749fbbae-5a3d-4686-8371-fa87a650f6c5 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:45.188664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749fbbae-5a3d-4686-8371-fa87a650f6c5 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:45.188664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749fbbae-5a3d-4686-8371-fa87a650f6c5 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:45.189976 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 738/1441] 127.0.0.1 () {54 vars in 917 bytes} [Wed Aug 24 15:40:45 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:40:45.195487 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29570bd1-3f30-4e81-9644-bba8a38e2dd5 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:45.195945 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29570bd1-3f30-4e81-9644-bba8a38e2dd5 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:45.196071 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29570bd1-3f30-4e81-9644-bba8a38e2dd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:45.236926 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:45.261677 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-29570bd1-3f30-4e81-9644-bba8a38e2dd5 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:40:45.262877 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-29570bd1-3f30-4e81-9644-bba8a38e2dd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:45.358004 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 704/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Aug 24 15:40:45 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:40:45.642640 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749fbbae-5a3d-4686-8371-fa87a650f6c5 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:40:45.644202 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-749fbbae-5a3d-4686-8371-fa87a650f6c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:45.747671 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749fbbae-5a3d-4686-8371-fa87a650f6c5 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:40:45.779910 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:45.824447 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c97bda-1bab-49fc-9761-4e5511d816ea service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:40:45.825079 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c97bda-1bab-49fc-9761-4e5511d816ea service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:45.825360 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c97bda-1bab-49fc-9761-4e5511d816ea service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:45.825636 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c97bda-1bab-49fc-9761-4e5511d816ea service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:45.827343 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-f9c97bda-1bab-49fc-9761-4e5511d816ea service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:45.944341 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9c97bda-1bab-49fc-9761-4e5511d816ea service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:40:45.945657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9c97bda-1bab-49fc-9761-4e5511d816ea service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:40:45.964657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 739/1443] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:40:45 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:40:48.067832 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941d73cd-2602-4aa3-aa83-54f840ca9895 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:48.068397 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941d73cd-2602-4aa3-aa83-54f840ca9895 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:48.068397 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941d73cd-2602-4aa3-aa83-54f840ca9895 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:48.069823 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 705/1444] 127.0.0.1 () {54 vars in 917 bytes} [Wed Aug 24 15:40:48 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:40:48.075093 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc32a39-af1a-4fc6-855e-0e6e6d92b671 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:48.075548 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc32a39-af1a-4fc6-855e-0e6e6d92b671 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:48.075710 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc32a39-af1a-4fc6-855e-0e6e6d92b671 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:48.144869 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-dcc32a39-af1a-4fc6-855e-0e6e6d92b671 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:40:48.146091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-dcc32a39-af1a-4fc6-855e-0e6e6d92b671 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:48.159583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:48.260071 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 740/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Aug 24 15:40:48 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:40:48.550129 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941d73cd-2602-4aa3-aa83-54f840ca9895 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:40:48.551724 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-941d73cd-2602-4aa3-aa83-54f840ca9895 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:48.656515 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941d73cd-2602-4aa3-aa83-54f840ca9895 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:40:48.669620 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:48.733495 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9470dc-07fc-432d-854c-f6e9b507705c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:40:48.734097 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9470dc-07fc-432d-854c-f6e9b507705c service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:48.734440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9470dc-07fc-432d-854c-f6e9b507705c service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:48.734720 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9470dc-07fc-432d-854c-f6e9b507705c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:48.736372 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-5a9470dc-07fc-432d-854c-f6e9b507705c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:48.851756 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a9470dc-07fc-432d-854c-f6e9b507705c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:40:48.853199 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a9470dc-07fc-432d-854c-f6e9b507705c service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:40:48.873317 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 706/1446] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:40:48 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:40:51.119575 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5476d8f4-2452-4662-a1da-2847baa23c82 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:51.120117 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5476d8f4-2452-4662-a1da-2847baa23c82 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:51.120117 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5476d8f4-2452-4662-a1da-2847baa23c82 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:51.121468 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 741/1447] 127.0.0.1 () {54 vars in 917 bytes} [Wed Aug 24 15:40:51 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:40:51.126979 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128e200d-5105-4da3-accc-2699b586f191 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:51.127429 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128e200d-5105-4da3-accc-2699b586f191 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:51.127542 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128e200d-5105-4da3-accc-2699b586f191 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:51.175191 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:51.193973 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-128e200d-5105-4da3-accc-2699b586f191 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:40:51.195997 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-128e200d-5105-4da3-accc-2699b586f191 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:51.291266 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 707/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Aug 24 15:40:51 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:40:51.577039 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5476d8f4-2452-4662-a1da-2847baa23c82 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:40:51.578590 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-5476d8f4-2452-4662-a1da-2847baa23c82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:51.686119 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5476d8f4-2452-4662-a1da-2847baa23c82 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:40:51.725585 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:51.759894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b4f35d-d60f-4c0d-9a56-30029a2ba270 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:40:51.760534 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b4f35d-d60f-4c0d-9a56-30029a2ba270 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:51.760816 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b4f35d-d60f-4c0d-9a56-30029a2ba270 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:51.761089 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b4f35d-d60f-4c0d-9a56-30029a2ba270 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:51.762744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-d7b4f35d-d60f-4c0d-9a56-30029a2ba270 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:51.876069 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b4f35d-d60f-4c0d-9a56-30029a2ba270 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:40:51.877527 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b4f35d-d60f-4c0d-9a56-30029a2ba270 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:40:51.895478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 742/1449] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:40:51 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:40:53.932278 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1981fd-a8ad-490f-8a53-5601ad4201e9 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:53.932810 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1981fd-a8ad-490f-8a53-5601ad4201e9 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:53.932810 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1981fd-a8ad-490f-8a53-5601ad4201e9 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:53.934163 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 708/1450] 127.0.0.1 () {54 vars in 917 bytes} [Wed Aug 24 15:40:53 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:40:53.939478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a1a356-d817-4619-a166-a097fcda47ec None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:53.939921 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a1a356-d817-4619-a166-a097fcda47ec None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:53.940065 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a1a356-d817-4619-a166-a097fcda47ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:54.006651 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-01a1a356-d817-4619-a166-a097fcda47ec None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:40:54.009036 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-01a1a356-d817-4619-a166-a097fcda47ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:54.029703 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:54.110184 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 743/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Aug 24 15:40:53 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:40:54.400353 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1981fd-a8ad-490f-8a53-5601ad4201e9 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:40:54.401869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-1b1981fd-a8ad-490f-8a53-5601ad4201e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:54.506421 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1981fd-a8ad-490f-8a53-5601ad4201e9 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:40:54.526729 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:54.581822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7479fe9a-e557-4531-beb9-ebb73bf7e267 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:40:54.582423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7479fe9a-e557-4531-beb9-ebb73bf7e267 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:54.582704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7479fe9a-e557-4531-beb9-ebb73bf7e267 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:54.582978 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7479fe9a-e557-4531-beb9-ebb73bf7e267 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:54.584618 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-7479fe9a-e557-4531-beb9-ebb73bf7e267 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:54.696013 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7479fe9a-e557-4531-beb9-ebb73bf7e267 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:40:54.697384 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7479fe9a-e557-4531-beb9-ebb73bf7e267 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:40:54.716915 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 709/1452] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:40:54 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:40:56.684867 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851e005f-61bc-40b5-857b-b9cf1adf2d79 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:56.685409 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851e005f-61bc-40b5-857b-b9cf1adf2d79 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:56.685409 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851e005f-61bc-40b5-857b-b9cf1adf2d79 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:56.686842 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 744/1453] 127.0.0.1 () {54 vars in 917 bytes} [Wed Aug 24 15:40:56 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:40:56.692264 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea3896d-ba48-4c7b-a54b-b729dedb3cd9 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:56.692693 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea3896d-ba48-4c7b-a54b-b729dedb3cd9 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:56.692852 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea3896d-ba48-4c7b-a54b-b729dedb3cd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:56.760441 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:56.763513 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-5ea3896d-ba48-4c7b-a54b-b729dedb3cd9 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:40:56.765200 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-5ea3896d-ba48-4c7b-a54b-b729dedb3cd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:56.874730 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 710/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Aug 24 15:40:56 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:40:57.157294 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851e005f-61bc-40b5-857b-b9cf1adf2d79 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:40:57.158851 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-851e005f-61bc-40b5-857b-b9cf1adf2d79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:57.262534 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851e005f-61bc-40b5-857b-b9cf1adf2d79 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:40:57.315008 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:57.341518 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f2d83c-5973-4126-8523-948cce895430 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:40:57.342123 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f2d83c-5973-4126-8523-948cce895430 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:57.342683 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f2d83c-5973-4126-8523-948cce895430 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:57.342955 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f2d83c-5973-4126-8523-948cce895430 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:57.344615 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-03f2d83c-5973-4126-8523-948cce895430 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:57.453941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f2d83c-5973-4126-8523-948cce895430 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:40:57.455289 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f2d83c-5973-4126-8523-948cce895430 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:40:57.473333 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 745/1455] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:40:57 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:40:59.477756 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32307f0a-b55e-4622-8eba-e5460fd798fd None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:59.478317 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32307f0a-b55e-4622-8eba-e5460fd798fd None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:59.478317 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32307f0a-b55e-4622-8eba-e5460fd798fd None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:59.479672 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 711/1456] 127.0.0.1 () {54 vars in 917 bytes} [Wed Aug 24 15:40:59 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:40:59.485145 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a994643-ebb1-46de-ab6f-4dfcd200c0bf None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:40:59.485571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a994643-ebb1-46de-ab6f-4dfcd200c0bf None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:40:59.485702 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a994643-ebb1-46de-ab6f-4dfcd200c0bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:40:59.557403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-6a994643-ebb1-46de-ab6f-4dfcd200c0bf None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:40:59.558673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-6a994643-ebb1-46de-ab6f-4dfcd200c0bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:40:59.604437 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:40:59.684299 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 746/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Aug 24 15:40:59 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:40:59.969504 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32307f0a-b55e-4622-8eba-e5460fd798fd None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:40:59.971034 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-32307f0a-b55e-4622-8eba-e5460fd798fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:00.079550 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32307f0a-b55e-4622-8eba-e5460fd798fd None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:41:00.116722 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:41:00.167440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e8af59-c510-4309-99af-df3a1b74604f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:41:00.168083 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e8af59-c510-4309-99af-df3a1b74604f service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:00.168365 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e8af59-c510-4309-99af-df3a1b74604f service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:00.168636 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e8af59-c510-4309-99af-df3a1b74604f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:00.170303 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-92e8af59-c510-4309-99af-df3a1b74604f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:00.286907 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e8af59-c510-4309-99af-df3a1b74604f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:41:00.288405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e8af59-c510-4309-99af-df3a1b74604f service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:41:00.308955 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 712/1458] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:40:59 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:41:02.271731 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1338c477-2b3f-42c0-8821-c811a152d2d4 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:02.272308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1338c477-2b3f-42c0-8821-c811a152d2d4 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:02.272308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1338c477-2b3f-42c0-8821-c811a152d2d4 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:02.274584 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 747/1459] 127.0.0.1 () {54 vars in 917 bytes} [Wed Aug 24 15:41:02 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:41:02.280094 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e8fa97-b85d-49c3-aace-76b743e8e1e0 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:02.280529 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e8fa97-b85d-49c3-aace-76b743e8e1e0 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:02.280646 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e8fa97-b85d-49c3-aace-76b743e8e1e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:02.352409 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-45e8fa97-b85d-49c3-aace-76b743e8e1e0 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:41:02.353618 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-45e8fa97-b85d-49c3-aace-76b743e8e1e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:02.361888 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:41:02.449958 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 713/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Aug 24 15:41:02 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:41:02.733537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1338c477-2b3f-42c0-8821-c811a152d2d4 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:41:02.735093 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-1338c477-2b3f-42c0-8821-c811a152d2d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:02.841345 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1338c477-2b3f-42c0-8821-c811a152d2d4 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:41:02.904389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:41:02.924855 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6ebdaf-76ae-41c5-b96f-ee7bbdb17df4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:41:02.925476 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6ebdaf-76ae-41c5-b96f-ee7bbdb17df4 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:02.925756 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6ebdaf-76ae-41c5-b96f-ee7bbdb17df4 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:02.926031 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6ebdaf-76ae-41c5-b96f-ee7bbdb17df4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:02.927697 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-4a6ebdaf-76ae-41c5-b96f-ee7bbdb17df4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:03.039612 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a6ebdaf-76ae-41c5-b96f-ee7bbdb17df4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:41:03.041007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a6ebdaf-76ae-41c5-b96f-ee7bbdb17df4 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:41:03.061346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 748/1461] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:41:02 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:41:05.029540 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4511241b-4627-4e7f-b6f4-270274f07c9e None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:05.030093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4511241b-4627-4e7f-b6f4-270274f07c9e None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:05.030093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4511241b-4627-4e7f-b6f4-270274f07c9e None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:05.031441 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 714/1462] 127.0.0.1 () {54 vars in 917 bytes} [Wed Aug 24 15:41:05 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:41:05.036901 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696c7b79-0563-4fa9-83e2-b3a451ca97c5 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:05.037305 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696c7b79-0563-4fa9-83e2-b3a451ca97c5 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:05.037457 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696c7b79-0563-4fa9-83e2-b3a451ca97c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:05.107894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-696c7b79-0563-4fa9-83e2-b3a451ca97c5 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:41:05.109033 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-696c7b79-0563-4fa9-83e2-b3a451ca97c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:05.147611 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:41:05.206058 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 749/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Aug 24 15:41:05 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:41:05.488782 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4511241b-4627-4e7f-b6f4-270274f07c9e None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:41:05.491391 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-4511241b-4627-4e7f-b6f4-270274f07c9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:05.609910 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4511241b-4627-4e7f-b6f4-270274f07c9e None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:41:05.648915 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:41:05.685197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51346e92-3c59-464d-91a8-8e47e5dd21fa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:41:05.685799 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51346e92-3c59-464d-91a8-8e47e5dd21fa service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:05.686078 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51346e92-3c59-464d-91a8-8e47e5dd21fa service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:05.686349 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51346e92-3c59-464d-91a8-8e47e5dd21fa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:05.687981 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-51346e92-3c59-464d-91a8-8e47e5dd21fa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:05.796909 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51346e92-3c59-464d-91a8-8e47e5dd21fa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:41:05.798383 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51346e92-3c59-464d-91a8-8e47e5dd21fa service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:41:05.817037 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 715/1464] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:41:05 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:41:07.832143 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f074a7-4dfc-43c3-a4f1-155cedfbf560 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:07.832772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f074a7-4dfc-43c3-a4f1-155cedfbf560 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:07.833080 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f074a7-4dfc-43c3-a4f1-155cedfbf560 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:07.834631 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 750/1465] 127.0.0.1 () {54 vars in 917 bytes} [Wed Aug 24 15:41:07 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:41:07.841352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01f086c-2c44-48f6-a583-9b2688aac124 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:07.841869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01f086c-2c44-48f6-a583-9b2688aac124 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:07.841988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01f086c-2c44-48f6-a583-9b2688aac124 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:07.926080 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-a01f086c-2c44-48f6-a583-9b2688aac124 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:41:07.927612 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-a01f086c-2c44-48f6-a583-9b2688aac124 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:07.948614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:41:08.043844 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 716/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Aug 24 15:41:07 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:41:08.347017 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f074a7-4dfc-43c3-a4f1-155cedfbf560 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:41:08.348704 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-54f074a7-4dfc-43c3-a4f1-155cedfbf560 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:08.575722 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f074a7-4dfc-43c3-a4f1-155cedfbf560 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:41:08.646829 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:41:08.654323 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca908af-621a-49b2-b7c3-f9d71c12dd8d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:41:08.654955 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca908af-621a-49b2-b7c3-f9d71c12dd8d service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:08.655232 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca908af-621a-49b2-b7c3-f9d71c12dd8d service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:08.655502 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca908af-621a-49b2-b7c3-f9d71c12dd8d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:08.657116 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-3ca908af-621a-49b2-b7c3-f9d71c12dd8d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:08.766615 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca908af-621a-49b2-b7c3-f9d71c12dd8d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:41:08.768087 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca908af-621a-49b2-b7c3-f9d71c12dd8d service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:41:08.785920 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 751/1467] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:41:08 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 440 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:41:10.733752 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a4c73a-3b88-47f1-a795-4608d268661a None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:10.734320 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a4c73a-3b88-47f1-a795-4608d268661a None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:10.734320 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a4c73a-3b88-47f1-a795-4608d268661a None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:10.735640 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 717/1468] 127.0.0.1 () {54 vars in 917 bytes} [Wed Aug 24 15:41:10 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:41:10.740886 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ccd91a-55b8-4989-beed-0740367be593 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:10.741403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ccd91a-55b8-4989-beed-0740367be593 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:10.741502 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ccd91a-55b8-4989-beed-0740367be593 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:10.809418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-f7ccd91a-55b8-4989-beed-0740367be593 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:41:10.810755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-f7ccd91a-55b8-4989-beed-0740367be593 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:10.860812 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:41:10.910952 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 752/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Aug 24 15:41:10 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:41:11.206463 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a4c73a-3b88-47f1-a795-4608d268661a None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:41:11.208018 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-64a4c73a-3b88-47f1-a795-4608d268661a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:11.309598 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a4c73a-3b88-47f1-a795-4608d268661a None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:41:11.359139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:41:11.388322 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f424a19-2f13-4ffe-b8d8-72652dd5d430 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:41:11.389586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f424a19-2f13-4ffe-b8d8-72652dd5d430 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:11.389920 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f424a19-2f13-4ffe-b8d8-72652dd5d430 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:11.390183 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f424a19-2f13-4ffe-b8d8-72652dd5d430 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:11.391874 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-5f424a19-2f13-4ffe-b8d8-72652dd5d430 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:11.503414 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f424a19-2f13-4ffe-b8d8-72652dd5d430 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:41:11.504869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f424a19-2f13-4ffe-b8d8-72652dd5d430 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:41:11.524153 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 718/1470] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:41:11 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:41:20.909120 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=22 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:41:21.521250 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=66 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:41:31.374141 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d7f46f-9d4d-4989-95d6-c4033559b7d6 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:31.374706 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d7f46f-9d4d-4989-95d6-c4033559b7d6 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:31.374706 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d7f46f-9d4d-4989-95d6-c4033559b7d6 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:31.376118 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 753/1471] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:41:31 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:41:31.381516 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9427b9ca-bde4-480c-bbd0-ac7c28af76fc None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:31.381967 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9427b9ca-bde4-480c-bbd0-ac7c28af76fc None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:31.382080 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9427b9ca-bde4-480c-bbd0-ac7c28af76fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:31.448781 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-9427b9ca-bde4-480c-bbd0-ac7c28af76fc None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:41:31.450000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-9427b9ca-bde4-480c-bbd0-ac7c28af76fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:31.553189 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 719/1472] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:41:31 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:41:31.849235 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a804f678-9f5a-40e6-aaad-7a3b666d29f9 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:31.849524 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a804f678-9f5a-40e6-aaad-7a3b666d29f9 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:31.849798 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a804f678-9f5a-40e6-aaad-7a3b666d29f9 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:31.851138 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 754/1473] 10.222.0.102 () {54 vars in 972 bytes} [Wed Aug 24 15:41:31 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:41:31.856564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00067aae-60a0-43d1-af40-b485041c009c None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:31.856858 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00067aae-60a0-43d1-af40-b485041c009c None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:31.857132 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00067aae-60a0-43d1-af40-b485041c009c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:31.935068 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-00067aae-60a0-43d1-af40-b485041c009c None None] MFA Rules not processed for user `8c6f06186c1a4626997db5a40c31cbb9`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:41:31.936314 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-00067aae-60a0-43d1-af40-b485041c009c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:31.957205 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:41:32.021034 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 720/1474] 10.222.0.102 () {58 vars in 1069 bytes} [Wed Aug 24 15:41:31 2022] POST /identity/v3/auth/tokens => generated 3130 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:41:32.029653 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a804f678-9f5a-40e6-aaad-7a3b666d29f9 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:41:32.031161 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-a804f678-9f5a-40e6-aaad-7a3b666d29f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:32.135943 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a804f678-9f5a-40e6-aaad-7a3b666d29f9 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:41:32.213864 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a9cc36d-8434-4b09-953d-71cd16dd9ab2 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c6f06186c1a4626997db5a40c31cbb9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:41:32.214539 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9cc36d-8434-4b09-953d-71cd16dd9ab2 service glance] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:32.214741 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9cc36d-8434-4b09-953d-71cd16dd9ab2 service glance] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:32.215011 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9cc36d-8434-4b09-953d-71cd16dd9ab2 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:32.216645 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-2a9cc36d-8434-4b09-953d-71cd16dd9ab2 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:32.270462 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:41:32.328412 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a9cc36d-8434-4b09-953d-71cd16dd9ab2 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:41:32.329868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a9cc36d-8434-4b09-953d-71cd16dd9ab2 service glance] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:41:32.348000 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 755/1475] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:41:32 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:41:34.386413 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098cb841-475d-49b7-946f-734478ba4fa2 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:34.386855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098cb841-475d-49b7-946f-734478ba4fa2 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:34.386950 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098cb841-475d-49b7-946f-734478ba4fa2 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:34.388381 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 721/1476] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:41:34 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:41:34.393830 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e82b94-357b-4006-b181-3e3532fe307a None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:34.394339 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e82b94-357b-4006-b181-3e3532fe307a None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:34.394424 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e82b94-357b-4006-b181-3e3532fe307a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:34.463813 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-e9e82b94-357b-4006-b181-3e3532fe307a None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:41:34.465005 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-e9e82b94-357b-4006-b181-3e3532fe307a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:34.559276 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 756/1477] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:41:34 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:41:34.838611 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098cb841-475d-49b7-946f-734478ba4fa2 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:41:34.840098 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-098cb841-475d-49b7-946f-734478ba4fa2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:34.972134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098cb841-475d-49b7-946f-734478ba4fa2 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:41:35.021470 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:41:35.069433 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9362d9f6-5fbe-4d9d-8a63-95cc38bd89a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:41:35.070032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9362d9f6-5fbe-4d9d-8a63-95cc38bd89a8 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:35.070371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9362d9f6-5fbe-4d9d-8a63-95cc38bd89a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:35.071223 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9362d9f6-5fbe-4d9d-8a63-95cc38bd89a8 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:35.071910 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9362d9f6-5fbe-4d9d-8a63-95cc38bd89a8 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:41:35.073659 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: WARNING py.warnings [None req-9362d9f6-5fbe-4d9d-8a63-95cc38bd89a8 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 24 15:41:35.073659 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: warnings.warn(msg) Aug 24 15:41:35.073659 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: Aug 24 15:41:35.074806 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9362d9f6-5fbe-4d9d-8a63-95cc38bd89a8 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:41:35.094084 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 722/1478] 10.222.0.102 () {56 vars in 1113 bytes} [Wed Aug 24 15:41:34 2022] GET /identity/v3/projects => generated 2562 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:41:37.074218 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bad9538-c324-4340-8f55-a851eadbf8f6 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:37.074737 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bad9538-c324-4340-8f55-a851eadbf8f6 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:37.074737 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bad9538-c324-4340-8f55-a851eadbf8f6 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:37.076176 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 757/1479] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:41:37 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:41:37.081641 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68943a34-4d69-45ec-bf01-107d60ec8595 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:37.082092 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68943a34-4d69-45ec-bf01-107d60ec8595 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:37.082208 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68943a34-4d69-45ec-bf01-107d60ec8595 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:37.152685 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-68943a34-4d69-45ec-bf01-107d60ec8595 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:41:37.152685 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-68943a34-4d69-45ec-bf01-107d60ec8595 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:37.249848 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 723/1480] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:41:37 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:41:37.551753 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bad9538-c324-4340-8f55-a851eadbf8f6 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:41:37.553669 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-6bad9538-c324-4340-8f55-a851eadbf8f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:37.564004 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:41:37.701318 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bad9538-c324-4340-8f55-a851eadbf8f6 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:41:37.785844 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:41:37.821036 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ef736e-9093-48ae-9f6e-36a4dd837d5c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:41:37.821823 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ef736e-9093-48ae-9f6e-36a4dd837d5c service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:37.822099 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ef736e-9093-48ae-9f6e-36a4dd837d5c service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:37.822371 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ef736e-9093-48ae-9f6e-36a4dd837d5c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:37.824096 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-b9ef736e-9093-48ae-9f6e-36a4dd837d5c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:37.916023 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ef736e-9093-48ae-9f6e-36a4dd837d5c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:41:37.917698 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ef736e-9093-48ae-9f6e-36a4dd837d5c service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:41:37.936673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 758/1481] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:41:37 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:41:40.085394 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396f7b80-31df-4364-9d71-edf85cc6c872 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:40.086037 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396f7b80-31df-4364-9d71-edf85cc6c872 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:40.086037 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396f7b80-31df-4364-9d71-edf85cc6c872 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:40.087985 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 724/1482] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:41:40 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:41:40.093753 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151673fc-dfe5-40aa-a463-99f49a790bfc None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:40.094172 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151673fc-dfe5-40aa-a463-99f49a790bfc None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:40.094286 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151673fc-dfe5-40aa-a463-99f49a790bfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:40.161876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-151673fc-dfe5-40aa-a463-99f49a790bfc None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:41:40.163091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-151673fc-dfe5-40aa-a463-99f49a790bfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:40.214625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:41:40.266849 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 759/1483] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:41:40 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:41:40.553349 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396f7b80-31df-4364-9d71-edf85cc6c872 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:41:40.555662 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-396f7b80-31df-4364-9d71-edf85cc6c872 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:40.579342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:41:40.657499 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396f7b80-31df-4364-9d71-edf85cc6c872 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:41:40.733731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f97789-2a73-4889-961f-f66b6f308ec5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:41:40.734340 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f97789-2a73-4889-961f-f66b6f308ec5 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:40.734630 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f97789-2a73-4889-961f-f66b6f308ec5 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:40.734900 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f97789-2a73-4889-961f-f66b6f308ec5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:40.736529 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-e4f97789-2a73-4889-961f-f66b6f308ec5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:40.817116 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:41:40.855751 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f97789-2a73-4889-961f-f66b6f308ec5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:41:40.857203 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f97789-2a73-4889-961f-f66b6f308ec5 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:41:40.876259 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 725/1484] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:41:40 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:41:42.866816 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371700c1-6fe0-4c1f-bb97-0225cc198df9 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:42.867366 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371700c1-6fe0-4c1f-bb97-0225cc198df9 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:42.867366 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371700c1-6fe0-4c1f-bb97-0225cc198df9 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:42.869319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 760/1485] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:41:42 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:41:42.874564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae276b7-98ce-46eb-aec7-0a7f94022d3f None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:42.875004 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae276b7-98ce-46eb-aec7-0a7f94022d3f None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:42.875122 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae276b7-98ce-46eb-aec7-0a7f94022d3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:42.947307 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-5ae276b7-98ce-46eb-aec7-0a7f94022d3f None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:41:42.948559 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-5ae276b7-98ce-46eb-aec7-0a7f94022d3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:43.060740 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 726/1486] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:41:42 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:41:43.346750 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371700c1-6fe0-4c1f-bb97-0225cc198df9 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:41:43.348327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-371700c1-6fe0-4c1f-bb97-0225cc198df9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:43.454099 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371700c1-6fe0-4c1f-bb97-0225cc198df9 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:41:43.524819 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:41:43.526739 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2dc290-1666-40bc-bc39-eb585aa64235 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:41:43.527377 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2dc290-1666-40bc-bc39-eb585aa64235 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:43.528101 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2dc290-1666-40bc-bc39-eb585aa64235 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:43.528568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2dc290-1666-40bc-bc39-eb585aa64235 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:43.530440 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-9a2dc290-1666-40bc-bc39-eb585aa64235 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:43.639354 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2dc290-1666-40bc-bc39-eb585aa64235 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:41:43.640824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2dc290-1666-40bc-bc39-eb585aa64235 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:41:43.658394 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 761/1487] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:41:43 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:41:45.673741 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52421bfd-7ea5-4e6f-83be-f9b2910d75fb None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:45.674250 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52421bfd-7ea5-4e6f-83be-f9b2910d75fb None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:45.674250 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52421bfd-7ea5-4e6f-83be-f9b2910d75fb None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:45.675563 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 727/1488] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:41:45 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:41:45.682004 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8457ad85-6448-4d73-99d1-4b4fecd0a7e4 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:45.682428 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8457ad85-6448-4d73-99d1-4b4fecd0a7e4 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:45.682565 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8457ad85-6448-4d73-99d1-4b4fecd0a7e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:45.749491 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-8457ad85-6448-4d73-99d1-4b4fecd0a7e4 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:41:45.750710 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-8457ad85-6448-4d73-99d1-4b4fecd0a7e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:45.823474 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:41:45.868482 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 762/1489] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:41:45 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:41:46.170584 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52421bfd-7ea5-4e6f-83be-f9b2910d75fb None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:41:46.172190 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-52421bfd-7ea5-4e6f-83be-f9b2910d75fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:46.208409 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:41:46.277707 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52421bfd-7ea5-4e6f-83be-f9b2910d75fb None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:41:46.350807 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc08f22-4b30-402e-b897-d930084c75f7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:41:46.351333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc08f22-4b30-402e-b897-d930084c75f7 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:46.351623 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc08f22-4b30-402e-b897-d930084c75f7 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:46.351955 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc08f22-4b30-402e-b897-d930084c75f7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:46.353706 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-ffc08f22-4b30-402e-b897-d930084c75f7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:46.441126 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:41:46.472116 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc08f22-4b30-402e-b897-d930084c75f7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:41:46.473584 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc08f22-4b30-402e-b897-d930084c75f7 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:41:46.492222 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 728/1490] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:41:46 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:41:48.702130 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df1272e-2406-4f21-81d1-d33f2525d78b None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:48.702697 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df1272e-2406-4f21-81d1-d33f2525d78b None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:48.702697 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df1272e-2406-4f21-81d1-d33f2525d78b None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:48.704061 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 763/1491] 10.222.0.102 () {54 vars in 932 bytes} [Wed Aug 24 15:41:48 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:41:48.709232 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e75284d-fc89-4a2a-934d-b096ae909469 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:48.709646 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e75284d-fc89-4a2a-934d-b096ae909469 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:48.709781 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e75284d-fc89-4a2a-934d-b096ae909469 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:48.775085 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-4e75284d-fc89-4a2a-934d-b096ae909469 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:41:48.776308 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-4e75284d-fc89-4a2a-934d-b096ae909469 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:48.875737 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 729/1492] 10.222.0.102 () {58 vars in 1029 bytes} [Wed Aug 24 15:41:48 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:41:49.165785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df1272e-2406-4f21-81d1-d33f2525d78b None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:41:49.167338 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-0df1272e-2406-4f21-81d1-d33f2525d78b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:49.275876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df1272e-2406-4f21-81d1-d33f2525d78b None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:41:49.350802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0938ba-e2a7-4ce2-bb99-47e4116cc67b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:41:49.351310 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0938ba-e2a7-4ce2-bb99-47e4116cc67b service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:41:49.351608 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0938ba-e2a7-4ce2-bb99-47e4116cc67b service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:41:49.351904 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0938ba-e2a7-4ce2-bb99-47e4116cc67b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:41:49.353490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-3f0938ba-e2a7-4ce2-bb99-47e4116cc67b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:41:49.362557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:41:49.468913 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0938ba-e2a7-4ce2-bb99-47e4116cc67b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:41:49.470293 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0938ba-e2a7-4ce2-bb99-47e4116cc67b service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:41:49.488167 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 764/1493] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:41:49 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:41:58.873851 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=82 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:41:59.485882 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=50 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:43:07.461733 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d000c3-5027-4198-90cd-932354b0cf48 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:43:07.462292 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d000c3-5027-4198-90cd-932354b0cf48 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:43:07.462292 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d000c3-5027-4198-90cd-932354b0cf48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:43:07.552854 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-06d000c3-5027-4198-90cd-932354b0cf48 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:43:07.555040 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-06d000c3-5027-4198-90cd-932354b0cf48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:43:07.677972 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 730/1494] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:43:07 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:43:07.688459 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9104c6a9-ef96-4f3f-b593-5e984bbb8a50 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:43:07.689015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9104c6a9-ef96-4f3f-b593-5e984bbb8a50 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:43:07.689015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9104c6a9-ef96-4f3f-b593-5e984bbb8a50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:43:07.773755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-9104c6a9-ef96-4f3f-b593-5e984bbb8a50 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:43:07.774915 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-9104c6a9-ef96-4f3f-b593-5e984bbb8a50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:43:07.905240 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 765/1495] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:43:07 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:43:07.910553 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d000c3-5027-4198-90cd-932354b0cf48 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:43:07.911869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-06d000c3-5027-4198-90cd-932354b0cf48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:43:07.924197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d000c3-5027-4198-90cd-932354b0cf48 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:43:07.999739 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:43:08.045611 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd479b3-3c20-47c3-9f14-5b52f7cc01d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:43:08.045611 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd479b3-3c20-47c3-9f14-5b52f7cc01d4 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:43:08.045611 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd479b3-3c20-47c3-9f14-5b52f7cc01d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:43:08.046020 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd479b3-3c20-47c3-9f14-5b52f7cc01d4 admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:43:08.047515 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd479b3-3c20-47c3-9f14-5b52f7cc01d4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:43:08.047853 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd479b3-3c20-47c3-9f14-5b52f7cc01d4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:43:08.048875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: WARNING py.warnings [None req-bdd479b3-3c20-47c3-9f14-5b52f7cc01d4 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 24 15:43:08.048875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: warnings.warn(msg) Aug 24 15:43:08.048875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: Aug 24 15:43:08.049841 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd479b3-3c20-47c3-9f14-5b52f7cc01d4 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:43:08.064412 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 731/1496] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:43:07 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:43:08.072979 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9104c6a9-ef96-4f3f-b593-5e984bbb8a50 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:43:08.082829 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9104c6a9-ef96-4f3f-b593-5e984bbb8a50 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:43:08.156879 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:43:08.198416 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270106c3-bd61-4c9a-b2bd-2d73470b8aaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:43:08.199569 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270106c3-bd61-4c9a-b2bd-2d73470b8aaf admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:43:08.199906 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270106c3-bd61-4c9a-b2bd-2d73470b8aaf admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:43:08.200148 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270106c3-bd61-4c9a-b2bd-2d73470b8aaf admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:43:08.200800 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270106c3-bd61-4c9a-b2bd-2d73470b8aaf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:43:08.202526 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: WARNING py.warnings [None req-270106c3-bd61-4c9a-b2bd-2d73470b8aaf admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 24 15:43:08.202526 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: warnings.warn(msg) Aug 24 15:43:08.202526 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: Aug 24 15:43:08.203741 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270106c3-bd61-4c9a-b2bd-2d73470b8aaf admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:43:08.260448 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 766/1497] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:43:08 2022] POST /identity/v3/projects => generated 380 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:43:08.265199 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd479b3-3c20-47c3-9f14-5b52f7cc01d4 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:43:08.266778 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-bdd479b3-3c20-47c3-9f14-5b52f7cc01d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:43:08.301009 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd479b3-3c20-47c3-9f14-5b52f7cc01d4 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:43:08.397640 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20adfa0-d6d1-490c-a83f-b212cfd48e2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:43:08.398235 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20adfa0-d6d1-490c-a83f-b212cfd48e2a admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:43:08.398511 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20adfa0-d6d1-490c-a83f-b212cfd48e2a admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:43:08.398868 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20adfa0-d6d1-490c-a83f-b212cfd48e2a admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:43:08.399413 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a20adfa0-d6d1-490c-a83f-b212cfd48e2a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:43:08.400454 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: WARNING py.warnings [None req-a20adfa0-d6d1-490c-a83f-b212cfd48e2a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 24 15:43:08.400454 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: warnings.warn(msg) Aug 24 15:43:08.400454 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: Aug 24 15:43:08.401649 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a20adfa0-d6d1-490c-a83f-b212cfd48e2a admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:43:08.445299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 732/1498] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:43:08 2022] POST /identity/v3/users => generated 291 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:43:08.449498 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270106c3-bd61-4c9a-b2bd-2d73470b8aaf admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:43:08.451022 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-270106c3-bd61-4c9a-b2bd-2d73470b8aaf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:43:08.513656 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270106c3-bd61-4c9a-b2bd-2d73470b8aaf admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:43:08.603043 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:43:08.614946 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f84a1d3-f45e-4c5e-8c6d-f02de14fa5ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:43:08.615554 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f84a1d3-f45e-4c5e-8c6d-f02de14fa5ca admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:43:08.615871 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f84a1d3-f45e-4c5e-8c6d-f02de14fa5ca admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:43:08.616145 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f84a1d3-f45e-4c5e-8c6d-f02de14fa5ca admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:43:08.616787 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f84a1d3-f45e-4c5e-8c6d-f02de14fa5ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:43:08.617799 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: WARNING py.warnings [None req-7f84a1d3-f45e-4c5e-8c6d-f02de14fa5ca admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 24 15:43:08.617799 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: warnings.warn(msg) Aug 24 15:43:08.617799 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: Aug 24 15:43:08.618685 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f84a1d3-f45e-4c5e-8c6d-f02de14fa5ca admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:43:08.630965 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 767/1499] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:43:08 2022] GET /identity/v3/roles => generated 1427 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 15:43:08.635029 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20adfa0-d6d1-490c-a83f-b212cfd48e2a admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:43:08.637245 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-a20adfa0-d6d1-490c-a83f-b212cfd48e2a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:43:08.648753 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20adfa0-d6d1-490c-a83f-b212cfd48e2a admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:43:08.675498 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:43:08.749433 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5303a4-a6a0-4fdc-bf92-3a0e1f2d37d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:43:08.750040 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5303a4-a6a0-4fdc-bf92-3a0e1f2d37d9 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:43:08.750323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5303a4-a6a0-4fdc-bf92-3a0e1f2d37d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:43:08.750598 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5303a4-a6a0-4fdc-bf92-3a0e1f2d37d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:43:08.751247 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5303a4-a6a0-4fdc-bf92-3a0e1f2d37d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:43:08.752290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: WARNING py.warnings [None req-fd5303a4-a6a0-4fdc-bf92-3a0e1f2d37d9 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 24 15:43:08.752290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: warnings.warn(msg) Aug 24 15:43:08.752290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: Aug 24 15:43:08.753185 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5303a4-a6a0-4fdc-bf92-3a0e1f2d37d9 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:43:08.766162 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 733/1500] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:43:08 2022] GET /identity/v3/roles => generated 1427 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:43:08.770680 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f84a1d3-f45e-4c5e-8c6d-f02de14fa5ca admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:43:08.772241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-7f84a1d3-f45e-4c5e-8c6d-f02de14fa5ca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:43:08.806593 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f84a1d3-f45e-4c5e-8c6d-f02de14fa5ca admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:43:08.906099 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711cc258-1ff0-45fd-b23f-dcf60af1db92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:43:08.906770 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711cc258-1ff0-45fd-b23f-dcf60af1db92 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:43:08.907055 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711cc258-1ff0-45fd-b23f-dcf60af1db92 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:43:08.907333 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711cc258-1ff0-45fd-b23f-dcf60af1db92 admin admin] PATH_INFO: `/v3/projects/40a164186dbb497dbb3d1292ddeaf055/users/d110859787a44416821f4dfd86166284/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:43:08.931195 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-711cc258-1ff0-45fd-b23f-dcf60af1db92 admin admin] RBAC: Authorizing `identity:create_grant(project_id=40a164186dbb497dbb3d1292ddeaf055, user_id=d110859787a44416821f4dfd86166284, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:43:08.932259 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: WARNING py.warnings [None req-711cc258-1ff0-45fd-b23f-dcf60af1db92 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 24 15:43:08.932259 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: warnings.warn(msg) Aug 24 15:43:08.932259 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: Aug 24 15:43:08.935073 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-711cc258-1ff0-45fd-b23f-dcf60af1db92 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:43:08.961146 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 768/1501] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:43:08 2022] PUT /identity/v3/projects/40a164186dbb497dbb3d1292ddeaf055/users/d110859787a44416821f4dfd86166284/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:43:08.973318 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e805a1f-c23d-480e-aadb-7b7697c1fb56 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:43:08.973607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e805a1f-c23d-480e-aadb-7b7697c1fb56 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:43:08.973887 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e805a1f-c23d-480e-aadb-7b7697c1fb56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:43:09.031831 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-1e805a1f-c23d-480e-aadb-7b7697c1fb56 None None] MFA Rules not processed for user `d110859787a44416821f4dfd86166284`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:43:09.033038 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-1e805a1f-c23d-480e-aadb-7b7697c1fb56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:43:09.111043 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 734/1502] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:43:08 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:43:09.124637 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32378023-f6fb-4611-8323-287f9ab1077e None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:43:09.124920 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32378023-f6fb-4611-8323-287f9ab1077e None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:43:09.125192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32378023-f6fb-4611-8323-287f9ab1077e None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:43:09.126567 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 769/1503] 10.222.0.102 () {54 vars in 969 bytes} [Wed Aug 24 15:43:09 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:43:09.131343 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d38cbfa-b73f-4b44-a469-59dc69fb87bb None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:43:09.131625 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d38cbfa-b73f-4b44-a469-59dc69fb87bb None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:43:09.131931 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d38cbfa-b73f-4b44-a469-59dc69fb87bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:43:09.163852 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:43:09.209786 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-0d38cbfa-b73f-4b44-a469-59dc69fb87bb None None] MFA Rules not processed for user `08fbd28d9593476b8753f8d64ee64f5b`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:43:09.210993 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-0d38cbfa-b73f-4b44-a469-59dc69fb87bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:43:09.285741 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 735/1504] 10.222.0.102 () {58 vars in 1066 bytes} [Wed Aug 24 15:43:09 2022] POST /identity/v3/auth/tokens => generated 3130 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:43:09.294594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32378023-f6fb-4611-8323-287f9ab1077e None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:43:09.296171 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-32378023-f6fb-4611-8323-287f9ab1077e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:43:09.350270 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:43:09.406023 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32378023-f6fb-4611-8323-287f9ab1077e None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:43:09.478616 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03890671-125c-464e-8f81-2080a1242fe1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:43:09.479219 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03890671-125c-464e-8f81-2080a1242fe1 service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:43:09.479496 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03890671-125c-464e-8f81-2080a1242fe1 service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:43:09.479806 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03890671-125c-464e-8f81-2080a1242fe1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:43:09.482007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-03890671-125c-464e-8f81-2080a1242fe1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:43:09.587149 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03890671-125c-464e-8f81-2080a1242fe1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:43:09.588684 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03890671-125c-464e-8f81-2080a1242fe1 service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:43:09.594323 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:43:09.609826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 770/1505] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:43:09 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:43:09.656072 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a7bdb8-d7e7-4409-b4e2-a2542147f13f None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:43:09.656515 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a7bdb8-d7e7-4409-b4e2-a2542147f13f None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:43:09.656625 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a7bdb8-d7e7-4409-b4e2-a2542147f13f None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:43:09.657865 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 736/1506] 10.222.0.102 () {52 vars in 835 bytes} [Wed Aug 24 15:43:09 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:43:09.661576 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03890671-125c-464e-8f81-2080a1242fe1 service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:43:09.663015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-03890671-125c-464e-8f81-2080a1242fe1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:43:09.767136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03890671-125c-464e-8f81-2080a1242fe1 service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:43:09.867463 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68720ef-0f44-436a-9324-3b45d09dceb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:43:09.868117 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68720ef-0f44-436a-9324-3b45d09dceb7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:43:09.868395 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68720ef-0f44-436a-9324-3b45d09dceb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:43:09.868670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68720ef-0f44-436a-9324-3b45d09dceb7 admin admin] PATH_INFO: `/v3/users/d110859787a44416821f4dfd86166284` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:43:09.881605 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f68720ef-0f44-436a-9324-3b45d09dceb7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d110859787a44416821f4dfd86166284)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:43:09.882640 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: WARNING py.warnings [None req-f68720ef-0f44-436a-9324-3b45d09dceb7 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 24 15:43:09.882640 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: warnings.warn(msg) Aug 24 15:43:09.882640 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: Aug 24 15:43:09.883885 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f68720ef-0f44-436a-9324-3b45d09dceb7 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:43:09.893833 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:43:09.955237 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-f68720ef-0f44-436a-9324-3b45d09dceb7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd110859787a44416821f4dfd86166284'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:43:09.955707 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-f68720ef-0f44-436a-9324-3b45d09dceb7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd110859787a44416821f4dfd86166284'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:43:09.956237 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-f68720ef-0f44-436a-9324-3b45d09dceb7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd110859787a44416821f4dfd86166284'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:43:09.967181 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-f68720ef-0f44-436a-9324-3b45d09dceb7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd110859787a44416821f4dfd86166284'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:43:09.990844 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-f68720ef-0f44-436a-9324-3b45d09dceb7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd110859787a44416821f4dfd86166284'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:43:10.010601 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 771/1507] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:43:09 2022] DELETE /identity/v3/users/d110859787a44416821f4dfd86166284 => generated 0 bytes in 350 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:43:10.015119 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a22e0c-234f-4629-8ea1-e751f5dcf691 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:43:10.015410 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a22e0c-234f-4629-8ea1-e751f5dcf691 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:43:10.015723 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a22e0c-234f-4629-8ea1-e751f5dcf691 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:43:10.073732 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-c9a22e0c-234f-4629-8ea1-e751f5dcf691 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:43:10.075461 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-c9a22e0c-234f-4629-8ea1-e751f5dcf691 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:43:10.160506 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:43:10.174972 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 737/1508] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:43:10 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:43:10.185010 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68720ef-0f44-436a-9324-3b45d09dceb7 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:43:10.186507 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-f68720ef-0f44-436a-9324-3b45d09dceb7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:43:10.287896 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68720ef-0f44-436a-9324-3b45d09dceb7 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:43:10.346693 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:43:10.370323 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309681e2-adac-404b-af77-2ff2f2fc36d6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:43:10.370963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309681e2-adac-404b-af77-2ff2f2fc36d6 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:43:10.371450 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309681e2-adac-404b-af77-2ff2f2fc36d6 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:43:10.371595 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309681e2-adac-404b-af77-2ff2f2fc36d6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:43:10.373122 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-309681e2-adac-404b-af77-2ff2f2fc36d6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:43:10.482278 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-309681e2-adac-404b-af77-2ff2f2fc36d6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:43:10.483770 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-309681e2-adac-404b-af77-2ff2f2fc36d6 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:43:10.503199 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 772/1509] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:43:10 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:43:10.933059 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a22e0c-234f-4629-8ea1-e751f5dcf691 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:43:10.934380 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-c9a22e0c-234f-4629-8ea1-e751f5dcf691 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:43:10.946174 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a22e0c-234f-4629-8ea1-e751f5dcf691 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:43:11.050481 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b1bbc2-9d41-4a29-a4de-2cae99ff1fa9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:43:11.051153 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b1bbc2-9d41-4a29-a4de-2cae99ff1fa9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:43:11.051467 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b1bbc2-9d41-4a29-a4de-2cae99ff1fa9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:43:11.051828 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b1bbc2-9d41-4a29-a4de-2cae99ff1fa9 admin admin] PATH_INFO: `/v3/projects/40a164186dbb497dbb3d1292ddeaf055` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:43:11.063383 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00b1bbc2-9d41-4a29-a4de-2cae99ff1fa9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=40a164186dbb497dbb3d1292ddeaf055)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:43:11.064456 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: WARNING py.warnings [None req-00b1bbc2-9d41-4a29-a4de-2cae99ff1fa9 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 24 15:43:11.064456 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: warnings.warn(msg) Aug 24 15:43:11.064456 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: Aug 24 15:43:11.065670 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00b1bbc2-9d41-4a29-a4de-2cae99ff1fa9 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:43:11.124905 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-00b1bbc2-9d41-4a29-a4de-2cae99ff1fa9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '40a164186dbb497dbb3d1292ddeaf055', 'request_id': 'req-00b1bbc2-9d41-4a29-a4de-2cae99ff1fa9'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:43:11.138579 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-00b1bbc2-9d41-4a29-a4de-2cae99ff1fa9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '40a164186dbb497dbb3d1292ddeaf055', 'request_id': 'req-00b1bbc2-9d41-4a29-a4de-2cae99ff1fa9'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:43:11.144588 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-00b1bbc2-9d41-4a29-a4de-2cae99ff1fa9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '40a164186dbb497dbb3d1292ddeaf055', 'request_id': 'req-00b1bbc2-9d41-4a29-a4de-2cae99ff1fa9'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:43:11.170831 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-00b1bbc2-9d41-4a29-a4de-2cae99ff1fa9 admin admin] The token cache is being invalidate because project 40a164186dbb497dbb3d1292ddeaf055 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:43:11.171120 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-00b1bbc2-9d41-4a29-a4de-2cae99ff1fa9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:43:11.171707 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-00b1bbc2-9d41-4a29-a4de-2cae99ff1fa9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:43:11.173120 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 738/1510] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:43:10 2022] DELETE /identity/v3/projects/40a164186dbb497dbb3d1292ddeaf055 => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:43:20.500860 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=62 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:43:21.168973 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:44:43.755124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaca285d-6b8d-441e-8945-3a2037f98855 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:44:43.755762 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaca285d-6b8d-441e-8945-3a2037f98855 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:44:43.755762 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaca285d-6b8d-441e-8945-3a2037f98855 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:44:43.757103 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 773/1511] 127.0.0.1 () {54 vars in 901 bytes} [Wed Aug 24 15:44:43 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:44:43.762002 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ffdd2e-0de7-4c50-970e-2efa0d3137d5 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:44:43.762440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ffdd2e-0de7-4c50-970e-2efa0d3137d5 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:44:43.762534 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ffdd2e-0de7-4c50-970e-2efa0d3137d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:44:43.845299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-c0ffdd2e-0de7-4c50-970e-2efa0d3137d5 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:44:43.847916 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-c0ffdd2e-0de7-4c50-970e-2efa0d3137d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:44:43.954196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 739/1512] 127.0.0.1 () {58 vars in 998 bytes} [Wed Aug 24 15:44:43 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:44:44.110543 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaca285d-6b8d-441e-8945-3a2037f98855 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:44:44.112774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-aaca285d-6b8d-441e-8945-3a2037f98855 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:44:44.234722 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaca285d-6b8d-441e-8945-3a2037f98855 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:44:44.240851 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be266e9-ea9f-4f59-b9b7-704c495eae2a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:44:44.241464 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be266e9-ea9f-4f59-b9b7-704c495eae2a service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:44:44.241747 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be266e9-ea9f-4f59-b9b7-704c495eae2a service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:44:44.242024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be266e9-ea9f-4f59-b9b7-704c495eae2a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:44:44.250550 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be266e9-ea9f-4f59-b9b7-704c495eae2a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:44:44.252221 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be266e9-ea9f-4f59-b9b7-704c495eae2a service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:44:44.261397 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 774/1513] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:44:44 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:44:45.119189 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1c2052-2113-4ef5-a66f-4553c392fd53 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:44:45.119747 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1c2052-2113-4ef5-a66f-4553c392fd53 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:44:45.119747 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1c2052-2113-4ef5-a66f-4553c392fd53 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:44:45.121043 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 740/1514] 127.0.0.1 () {54 vars in 901 bytes} [Wed Aug 24 15:44:45 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (2 switches on core 0) Aug 24 15:44:45.126201 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88aa579f-7ad7-44c2-979f-44535eff68b6 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:44:45.127738 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88aa579f-7ad7-44c2-979f-44535eff68b6 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:44:45.127738 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88aa579f-7ad7-44c2-979f-44535eff68b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:44:45.148488 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-88aa579f-7ad7-44c2-979f-44535eff68b6 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:44:45.149695 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-88aa579f-7ad7-44c2-979f-44535eff68b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:44:45.157527 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 775/1515] 127.0.0.1 () {58 vars in 998 bytes} [Wed Aug 24 15:44:45 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:44:45.321431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1c2052-2113-4ef5-a66f-4553c392fd53 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:44:45.336220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1c2052-2113-4ef5-a66f-4553c392fd53 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:44:45.341558 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af09f5c-03b8-44ac-ab2e-c87497d7a514 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:44:45.342139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af09f5c-03b8-44ac-ab2e-c87497d7a514 service nova] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:44:45.342418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af09f5c-03b8-44ac-ab2e-c87497d7a514 service nova] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:44:45.342689 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af09f5c-03b8-44ac-ab2e-c87497d7a514 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:44:45.350682 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0af09f5c-03b8-44ac-ab2e-c87497d7a514 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:44:45.352289 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0af09f5c-03b8-44ac-ab2e-c87497d7a514 service nova] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:44:45.361615 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 741/1516] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:44:45 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:44:46.364898 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7118eb90-0fcd-4e93-a2b9-46362a50458c None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:44:46.365471 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7118eb90-0fcd-4e93-a2b9-46362a50458c None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:44:46.365471 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7118eb90-0fcd-4e93-a2b9-46362a50458c None None] PATH_INFO: `/v3` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:44:46.366752 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 776/1517] 10.222.0.62 () {54 vars in 929 bytes} [Wed Aug 24 15:44:46 2022] GET /identity/v3 => generated 256 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:44:46.384888 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307df39d-552e-4823-bcd4-e1a1badb7917 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:44:46.385349 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307df39d-552e-4823-bcd4-e1a1badb7917 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:44:46.385459 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307df39d-552e-4823-bcd4-e1a1badb7917 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:44:46.443840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-307df39d-552e-4823-bcd4-e1a1badb7917 None None] MFA Rules not processed for user `0ee0002a76c3471f819b37fef510ef99`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:44:46.445048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-307df39d-552e-4823-bcd4-e1a1badb7917 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:44:46.461975 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:44:46.517661 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 742/1518] 10.222.0.62 () {58 vars in 1019 bytes} [Wed Aug 24 15:44:46 2022] POST /identity/v3/auth/tokens => generated 3318 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:44:50.516829 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8132e5f1-f4be-4db7-924e-a1cd874b9ce5 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:44:50.517344 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8132e5f1-f4be-4db7-924e-a1cd874b9ce5 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:44:50.517344 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8132e5f1-f4be-4db7-924e-a1cd874b9ce5 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:44:50.519313 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 777/1519] 10.222.0.102 () {54 vars in 966 bytes} [Wed Aug 24 15:44:50 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:44:50.524694 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e692b17-96b1-4f2e-a5aa-9d7cf7198ebf None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:44:50.525147 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e692b17-96b1-4f2e-a5aa-9d7cf7198ebf None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:44:50.525378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e692b17-96b1-4f2e-a5aa-9d7cf7198ebf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:44:50.544972 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-5e692b17-96b1-4f2e-a5aa-9d7cf7198ebf None None] MFA Rules not processed for user `0ee0002a76c3471f819b37fef510ef99`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:44:50.546166 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-5e692b17-96b1-4f2e-a5aa-9d7cf7198ebf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:44:50.554696 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 743/1520] 10.222.0.102 () {58 vars in 1063 bytes} [Wed Aug 24 15:44:50 2022] POST /identity/v3/auth/tokens => generated 3318 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:44:50.562441 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8132e5f1-f4be-4db7-924e-a1cd874b9ce5 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:44:50.576499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8132e5f1-f4be-4db7-924e-a1cd874b9ce5 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:44:50.581945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a8182c-8852-40fc-becb-64a44cdf7383 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0ee0002a76c3471f819b37fef510ef99', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:44:50.582525 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a8182c-8852-40fc-becb-64a44cdf7383 service placement] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:44:50.582816 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a8182c-8852-40fc-becb-64a44cdf7383 service placement] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:44:50.583079 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a8182c-8852-40fc-becb-64a44cdf7383 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:44:50.591263 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1a8182c-8852-40fc-becb-64a44cdf7383 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:44:50.592904 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1a8182c-8852-40fc-becb-64a44cdf7383 service placement] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:44:50.602555 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 778/1521] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:44:50 2022] GET /identity/v3/auth/tokens => generated 3318 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:44:56.623342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317cbaea-3f63-4d06-8d1a-eca4ef68fdfd None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:44:56.623959 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317cbaea-3f63-4d06-8d1a-eca4ef68fdfd None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:44:56.623959 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317cbaea-3f63-4d06-8d1a-eca4ef68fdfd None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:44:56.625279 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 744/1522] 127.0.0.1 () {54 vars in 901 bytes} [Wed Aug 24 15:44:56 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:44:56.630655 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5719d35e-b93f-4ed1-84ff-1e16e55bc1ec None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:44:56.631049 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5719d35e-b93f-4ed1-84ff-1e16e55bc1ec None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:44:56.631229 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5719d35e-b93f-4ed1-84ff-1e16e55bc1ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:44:56.651134 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-5719d35e-b93f-4ed1-84ff-1e16e55bc1ec None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:44:56.652364 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-5719d35e-b93f-4ed1-84ff-1e16e55bc1ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:44:56.660128 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 779/1523] 127.0.0.1 () {58 vars in 998 bytes} [Wed Aug 24 15:44:56 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:44:56.815473 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317cbaea-3f63-4d06-8d1a-eca4ef68fdfd None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:44:56.831248 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317cbaea-3f63-4d06-8d1a-eca4ef68fdfd None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:44:56.837377 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cae3e4-827e-426a-808a-c613f65f6285 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:44:56.837975 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cae3e4-827e-426a-808a-c613f65f6285 service nova] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:44:56.838258 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cae3e4-827e-426a-808a-c613f65f6285 service nova] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:44:56.838538 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cae3e4-827e-426a-808a-c613f65f6285 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:44:56.847488 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7cae3e4-827e-426a-808a-c613f65f6285 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:44:56.849988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7cae3e4-827e-426a-808a-c613f65f6285 service nova] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:44:56.860250 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 745/1524] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:44:56 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:44:57.681546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fc9a0e-6051-4ad9-ab43-3932dfbea63b None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:44:57.682135 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fc9a0e-6051-4ad9-ab43-3932dfbea63b None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:44:57.682135 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fc9a0e-6051-4ad9-ab43-3932dfbea63b None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:44:57.683472 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 780/1525] 127.0.0.1 () {54 vars in 901 bytes} [Wed Aug 24 15:44:57 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:44:57.688453 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567affc6-7f7b-49f7-8a1f-19dda40e29a3 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:44:57.688879 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567affc6-7f7b-49f7-8a1f-19dda40e29a3 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:44:57.688990 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567affc6-7f7b-49f7-8a1f-19dda40e29a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:44:57.710911 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-567affc6-7f7b-49f7-8a1f-19dda40e29a3 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:44:57.712150 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-567affc6-7f7b-49f7-8a1f-19dda40e29a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:44:57.720061 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 746/1526] 127.0.0.1 () {58 vars in 998 bytes} [Wed Aug 24 15:44:57 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:44:57.872871 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fc9a0e-6051-4ad9-ab43-3932dfbea63b None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:44:57.887705 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fc9a0e-6051-4ad9-ab43-3932dfbea63b None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:44:57.893050 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01dc335-d832-4d64-91b4-3c9dca097dd1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:44:57.893637 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01dc335-d832-4d64-91b4-3c9dca097dd1 service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:44:57.893915 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01dc335-d832-4d64-91b4-3c9dca097dd1 service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:44:57.894186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01dc335-d832-4d64-91b4-3c9dca097dd1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:44:57.903278 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01dc335-d832-4d64-91b4-3c9dca097dd1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:44:57.904898 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01dc335-d832-4d64-91b4-3c9dca097dd1 service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:44:57.914783 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 781/1527] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:44:57 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:45:01.858456 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c2b0b8-9b18-4a99-a341-327522257a1f None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:45:01.858959 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c2b0b8-9b18-4a99-a341-327522257a1f None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:45:01.859053 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c2b0b8-9b18-4a99-a341-327522257a1f None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:45:01.860439 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 747/1528] 127.0.0.1 () {54 vars in 917 bytes} [Wed Aug 24 15:45:01 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:45:01.866380 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92da0a24-47cd-4b93-bca9-c9102016f7cb None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:45:01.866769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92da0a24-47cd-4b93-bca9-c9102016f7cb None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:45:01.866941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92da0a24-47cd-4b93-bca9-c9102016f7cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:45:01.887140 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-92da0a24-47cd-4b93-bca9-c9102016f7cb None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:45:01.888376 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-92da0a24-47cd-4b93-bca9-c9102016f7cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:45:01.897583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 782/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Aug 24 15:45:01 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:45:02.200859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c2b0b8-9b18-4a99-a341-327522257a1f None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:45:02.202684 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-c3c2b0b8-9b18-4a99-a341-327522257a1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:45:02.286800 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c2b0b8-9b18-4a99-a341-327522257a1f None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:45:02.290647 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a61cdaf3-b05b-46f1-9c7d-b34054e75cce service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:45:02.291243 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61cdaf3-b05b-46f1-9c7d-b34054e75cce service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:45:02.291522 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61cdaf3-b05b-46f1-9c7d-b34054e75cce service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:45:02.291819 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61cdaf3-b05b-46f1-9c7d-b34054e75cce service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:45:02.301262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a61cdaf3-b05b-46f1-9c7d-b34054e75cce service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:45:02.302729 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a61cdaf3-b05b-46f1-9c7d-b34054e75cce service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:45:02.312074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 748/1530] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:45:02 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:45:04.209916 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f947da-71c0-4f3f-9e0a-aecbb66f5e97 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:45:04.210484 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f947da-71c0-4f3f-9e0a-aecbb66f5e97 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:45:04.210484 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f947da-71c0-4f3f-9e0a-aecbb66f5e97 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:45:04.211900 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 783/1531] 127.0.0.1 () {54 vars in 917 bytes} [Wed Aug 24 15:45:04 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:45:04.217037 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d70a112-4d9b-498e-a423-b3f48047ed21 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:45:04.217489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d70a112-4d9b-498e-a423-b3f48047ed21 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:45:04.217589 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d70a112-4d9b-498e-a423-b3f48047ed21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:45:04.237267 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-0d70a112-4d9b-498e-a423-b3f48047ed21 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:45:04.238483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-0d70a112-4d9b-498e-a423-b3f48047ed21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:45:04.247191 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 749/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Aug 24 15:45:04 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:45:04.536726 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f947da-71c0-4f3f-9e0a-aecbb66f5e97 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:45:04.547467 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f947da-71c0-4f3f-9e0a-aecbb66f5e97 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:45:04.551672 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5544c5f9-e463-4af1-9102-245d62eeec5a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:45:04.552270 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5544c5f9-e463-4af1-9102-245d62eeec5a service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:45:04.552557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5544c5f9-e463-4af1-9102-245d62eeec5a service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:45:04.552835 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5544c5f9-e463-4af1-9102-245d62eeec5a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:45:04.562171 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5544c5f9-e463-4af1-9102-245d62eeec5a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:45:04.563617 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5544c5f9-e463-4af1-9102-245d62eeec5a service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:45:04.573587 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 784/1533] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:45:04 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:45:08.129521 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe32e225-ccd8-481d-a035-a1b3d4082d2a None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:45:08.130045 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe32e225-ccd8-481d-a035-a1b3d4082d2a None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:45:08.130163 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe32e225-ccd8-481d-a035-a1b3d4082d2a None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:45:08.131431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 750/1534] 127.0.0.1 () {54 vars in 917 bytes} [Wed Aug 24 15:45:08 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:45:08.136600 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6750caea-218a-45fb-b8d5-1cf3d2e6ca05 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:45:08.136993 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6750caea-218a-45fb-b8d5-1cf3d2e6ca05 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:45:08.137161 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6750caea-218a-45fb-b8d5-1cf3d2e6ca05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:45:08.157183 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-6750caea-218a-45fb-b8d5-1cf3d2e6ca05 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:45:08.158388 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-6750caea-218a-45fb-b8d5-1cf3d2e6ca05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:45:08.167269 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 785/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Aug 24 15:45:08 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:45:08.742326 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe32e225-ccd8-481d-a035-a1b3d4082d2a None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:45:08.744028 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-fe32e225-ccd8-481d-a035-a1b3d4082d2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:45:08.809310 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:45:08.823534 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe32e225-ccd8-481d-a035-a1b3d4082d2a None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:45:08.827217 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d81da3-c8cd-411a-9bf0-cb5b59f2ed72 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:45:08.827852 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d81da3-c8cd-411a-9bf0-cb5b59f2ed72 service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:45:08.828138 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d81da3-c8cd-411a-9bf0-cb5b59f2ed72 service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:45:08.828413 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d81da3-c8cd-411a-9bf0-cb5b59f2ed72 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:45:08.839539 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d81da3-c8cd-411a-9bf0-cb5b59f2ed72 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:45:08.841013 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d81da3-c8cd-411a-9bf0-cb5b59f2ed72 service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:45:08.852001 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 751/1536] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:45:08 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:45:10.875505 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d3388e-2477-4ed8-b7aa-7dfda5bdb278 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:45:10.876100 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d3388e-2477-4ed8-b7aa-7dfda5bdb278 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:45:10.876416 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d3388e-2477-4ed8-b7aa-7dfda5bdb278 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:45:10.877812 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 786/1537] 127.0.0.1 () {54 vars in 917 bytes} [Wed Aug 24 15:45:10 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:45:10.883492 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2057943-69ca-4108-b0bb-3d5e6282e25b None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:45:10.884005 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2057943-69ca-4108-b0bb-3d5e6282e25b None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:45:10.884124 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2057943-69ca-4108-b0bb-3d5e6282e25b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:45:10.906286 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-f2057943-69ca-4108-b0bb-3d5e6282e25b None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:45:10.908005 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-f2057943-69ca-4108-b0bb-3d5e6282e25b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:45:10.919906 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 752/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Aug 24 15:45:10 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:45:11.497416 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d3388e-2477-4ed8-b7aa-7dfda5bdb278 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:45:11.499086 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-a4d3388e-2477-4ed8-b7aa-7dfda5bdb278 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:45:11.506708 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=84 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:45:11.516790 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d3388e-2477-4ed8-b7aa-7dfda5bdb278 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:45:11.520399 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627764ee-7801-45c6-9c00-de72014c8b0e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:45:11.521001 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627764ee-7801-45c6-9c00-de72014c8b0e service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:45:11.521290 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627764ee-7801-45c6-9c00-de72014c8b0e service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:45:11.521563 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627764ee-7801-45c6-9c00-de72014c8b0e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:45:11.530985 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627764ee-7801-45c6-9c00-de72014c8b0e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:45:11.532468 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627764ee-7801-45c6-9c00-de72014c8b0e service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:45:11.548242 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 787/1539] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:45:11 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:45:20.900784 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=11 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:45:21.542133 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=5 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:46:42.489381 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7547c2-568e-418e-bc4e-3b56853913f9 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:42.490486 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7547c2-568e-418e-bc4e-3b56853913f9 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:42.490950 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7547c2-568e-418e-bc4e-3b56853913f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:42.520739 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-1d7547c2-568e-418e-bc4e-3b56853913f9 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:42.522290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-1d7547c2-568e-418e-bc4e-3b56853913f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:42.530998 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 753/1540] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:42 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:42.541618 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81b3ee1-59f2-4b29-8e5f-35d2b941e7f7 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:42.542230 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81b3ee1-59f2-4b29-8e5f-35d2b941e7f7 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:42.542230 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81b3ee1-59f2-4b29-8e5f-35d2b941e7f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:42.580583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-b81b3ee1-59f2-4b29-8e5f-35d2b941e7f7 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:42.581858 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-b81b3ee1-59f2-4b29-8e5f-35d2b941e7f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:42.595386 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 788/1541] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:42 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:42.600647 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7547c2-568e-418e-bc4e-3b56853913f9 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:42.615338 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7547c2-568e-418e-bc4e-3b56853913f9 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:42.626647 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244c455e-d2c3-463b-b153-1fd39bd62153 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:42.627687 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244c455e-d2c3-463b-b153-1fd39bd62153 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:42.628190 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244c455e-d2c3-463b-b153-1fd39bd62153 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:42.628651 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244c455e-d2c3-463b-b153-1fd39bd62153 admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:42.629674 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244c455e-d2c3-463b-b153-1fd39bd62153 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:46:42.630209 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244c455e-d2c3-463b-b153-1fd39bd62153 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:42.632139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244c455e-d2c3-463b-b153-1fd39bd62153 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:42.654388 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 754/1542] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:46:42 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:46:42.663896 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81b3ee1-59f2-4b29-8e5f-35d2b941e7f7 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:42.683297 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81b3ee1-59f2-4b29-8e5f-35d2b941e7f7 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:42.692531 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b03ddec-cf56-445c-9d65-a48edb18db6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:42.693263 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b03ddec-cf56-445c-9d65-a48edb18db6d admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:42.693549 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b03ddec-cf56-445c-9d65-a48edb18db6d admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:42.693986 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b03ddec-cf56-445c-9d65-a48edb18db6d admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:42.694666 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b03ddec-cf56-445c-9d65-a48edb18db6d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:42.697504 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b03ddec-cf56-445c-9d65-a48edb18db6d admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:42.768696 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 789/1543] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:46:42 2022] POST /identity/v3/projects => generated 392 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:46:42.773092 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244c455e-d2c3-463b-b153-1fd39bd62153 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:42.788655 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244c455e-d2c3-463b-b153-1fd39bd62153 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:42.793940 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080338df-4263-418f-af7b-140bdaeb2ac0 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:42.794286 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080338df-4263-418f-af7b-140bdaeb2ac0 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:42.794494 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080338df-4263-418f-af7b-140bdaeb2ac0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:42.824895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-080338df-4263-418f-af7b-140bdaeb2ac0 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:42.826118 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-080338df-4263-418f-af7b-140bdaeb2ac0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:42.926456 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c68b97-3125-45fa-958e-e86fea107e0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:42.927490 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c68b97-3125-45fa-958e-e86fea107e0c admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:42.928011 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c68b97-3125-45fa-958e-e86fea107e0c admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:42.928835 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c68b97-3125-45fa-958e-e86fea107e0c admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:42.929626 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c68b97-3125-45fa-958e-e86fea107e0c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:42.942305 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c68b97-3125-45fa-958e-e86fea107e0c admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:42.948531 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 790/1544] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:42 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:42.949869 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c9ba0e-1e27-43eb-b4a9-77a1da3c61f2 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:42.950149 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c9ba0e-1e27-43eb-b4a9-77a1da3c61f2 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:42.950552 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c9ba0e-1e27-43eb-b4a9-77a1da3c61f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:42.998750 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-76c9ba0e-1e27-43eb-b4a9-77a1da3c61f2 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:42.999994 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-76c9ba0e-1e27-43eb-b4a9-77a1da3c61f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:43.008759 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 791/1545] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:42 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:43.010097 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01782a0-d719-4173-bea0-45fcea67b62c None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:43.010384 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01782a0-d719-4173-bea0-45fcea67b62c None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:43.010656 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01782a0-d719-4173-bea0-45fcea67b62c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:43.016945 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 755/1546] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:46:42 2022] POST /identity/v3/users => generated 297 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:46:43.025173 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17468c65-4112-4d51-a8d4-5ff63dd6eb99 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:43.026404 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17468c65-4112-4d51-a8d4-5ff63dd6eb99 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:43.026404 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17468c65-4112-4d51-a8d4-5ff63dd6eb99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:43.049616 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-d01782a0-d719-4173-bea0-45fcea67b62c None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:43.051831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-d01782a0-d719-4173-bea0-45fcea67b62c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:43.062879 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-17468c65-4112-4d51-a8d4-5ff63dd6eb99 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:43.065486 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-17468c65-4112-4d51-a8d4-5ff63dd6eb99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:43.073215 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 792/1547] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:43 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:43.074439 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01782a0-d719-4173-bea0-45fcea67b62c None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:43.089842 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 756/1548] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:43 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 72 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:43.091450 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb47fa18-2b09-4f43-af3c-b9fe966169b1 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:43.093265 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb47fa18-2b09-4f43-af3c-b9fe966169b1 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:43.094364 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb47fa18-2b09-4f43-af3c-b9fe966169b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:43.094932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01782a0-d719-4173-bea0-45fcea67b62c None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:43.108044 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c168588-fdbb-4003-88e0-101952dc6e30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:43.109516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c168588-fdbb-4003-88e0-101952dc6e30 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:43.110022 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c168588-fdbb-4003-88e0-101952dc6e30 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:43.110515 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c168588-fdbb-4003-88e0-101952dc6e30 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:43.111379 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c168588-fdbb-4003-88e0-101952dc6e30 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:43.113276 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c168588-fdbb-4003-88e0-101952dc6e30 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:43.128383 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-bb47fa18-2b09-4f43-af3c-b9fe966169b1 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:43.131162 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-bb47fa18-2b09-4f43-af3c-b9fe966169b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:43.137873 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 793/1549] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:46:43 2022] GET /identity/v3/roles => generated 1427 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:46:43.140006 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 757/1550] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:43 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:43.140537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c3b01d-0725-47a9-9c2a-f20ee4386fce None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:43.141287 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c3b01d-0725-47a9-9c2a-f20ee4386fce None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:43.141645 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb47fa18-2b09-4f43-af3c-b9fe966169b1 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:43.141765 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c3b01d-0725-47a9-9c2a-f20ee4386fce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:43.155724 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb47fa18-2b09-4f43-af3c-b9fe966169b1 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:43.169939 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10370e68-239f-4e78-b095-ef8f06d75ea2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:43.170887 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10370e68-239f-4e78-b095-ef8f06d75ea2 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:43.171578 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10370e68-239f-4e78-b095-ef8f06d75ea2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:43.172110 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10370e68-239f-4e78-b095-ef8f06d75ea2 admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:43.173088 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10370e68-239f-4e78-b095-ef8f06d75ea2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:46:43.173604 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10370e68-239f-4e78-b095-ef8f06d75ea2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:43.174433 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-40c3b01d-0725-47a9-9c2a-f20ee4386fce None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:43.175753 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10370e68-239f-4e78-b095-ef8f06d75ea2 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:43.177086 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-40c3b01d-0725-47a9-9c2a-f20ee4386fce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:43.191148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 758/1551] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:46:43 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:46:43.192541 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 794/1552] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:43 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:43.198332 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ffbc2e-0d70-4758-9a47-45178ef9db18 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:43.198511 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f082c609-2a1b-46e5-ae02-5484b65912ac None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:43.198511 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f082c609-2a1b-46e5-ae02-5484b65912ac None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:43.198511 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f082c609-2a1b-46e5-ae02-5484b65912ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:43.199735 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ffbc2e-0d70-4758-9a47-45178ef9db18 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:43.200244 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ffbc2e-0d70-4758-9a47-45178ef9db18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:43.226932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-c4ffbc2e-0d70-4758-9a47-45178ef9db18 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:43.228629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-c4ffbc2e-0d70-4758-9a47-45178ef9db18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:43.238108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 759/1553] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:43 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:43.239152 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ffbc2e-0d70-4758-9a47-45178ef9db18 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:43.241888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-f082c609-2a1b-46e5-ae02-5484b65912ac None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:43.243734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-f082c609-2a1b-46e5-ae02-5484b65912ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:43.249364 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ffbc2e-0d70-4758-9a47-45178ef9db18 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:43.252207 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 795/1554] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:43 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:43.253164 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f082c609-2a1b-46e5-ae02-5484b65912ac None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:43.253904 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909dba81-2ab1-4012-b95e-0efbd33e2f10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:43.254497 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909dba81-2ab1-4012-b95e-0efbd33e2f10 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:43.254774 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909dba81-2ab1-4012-b95e-0efbd33e2f10 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:43.255044 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909dba81-2ab1-4012-b95e-0efbd33e2f10 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:43.255951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909dba81-2ab1-4012-b95e-0efbd33e2f10 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:43.257540 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909dba81-2ab1-4012-b95e-0efbd33e2f10 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:43.264247 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f082c609-2a1b-46e5-ae02-5484b65912ac None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:43.268780 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 760/1555] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:46:43 2022] GET /identity/v3/roles => generated 1427 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:46:43.268962 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2455fd2-5a97-41a0-bdd4-5ac5a667a857 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:43.269454 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909dba81-2ab1-4012-b95e-0efbd33e2f10 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:43.269631 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2455fd2-5a97-41a0-bdd4-5ac5a667a857 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:43.269920 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2455fd2-5a97-41a0-bdd4-5ac5a667a857 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:43.270221 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2455fd2-5a97-41a0-bdd4-5ac5a667a857 admin admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:43.270846 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2455fd2-5a97-41a0-bdd4-5ac5a667a857 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:46:43.271261 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2455fd2-5a97-41a0-bdd4-5ac5a667a857 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:43.272423 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: WARNING py.warnings [None req-c2455fd2-5a97-41a0-bdd4-5ac5a667a857 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 24 15:46:43.272423 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: warnings.warn(msg) Aug 24 15:46:43.272423 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: Aug 24 15:46:43.273420 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2455fd2-5a97-41a0-bdd4-5ac5a667a857 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:43.280056 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909dba81-2ab1-4012-b95e-0efbd33e2f10 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:43.284602 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7096650f-bda6-4133-844c-64ef7312a62d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:43.285210 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7096650f-bda6-4133-844c-64ef7312a62d admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:43.285483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7096650f-bda6-4133-844c-64ef7312a62d admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:43.285752 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7096650f-bda6-4133-844c-64ef7312a62d admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:43.293678 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 796/1556] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:46:43 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:46:43.293850 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7096650f-bda6-4133-844c-64ef7312a62d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:43.293850 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: WARNING py.warnings [None req-7096650f-bda6-4133-844c-64ef7312a62d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 24 15:46:43.293850 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: warnings.warn(msg) Aug 24 15:46:43.293850 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: Aug 24 15:46:43.294205 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf2eff1-99a5-41d8-97a6-00ff95be65ef None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:43.294205 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf2eff1-99a5-41d8-97a6-00ff95be65ef None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:43.294205 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf2eff1-99a5-41d8-97a6-00ff95be65ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:43.294498 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7096650f-bda6-4133-844c-64ef7312a62d admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:43.308125 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-8cf2eff1-99a5-41d8-97a6-00ff95be65ef None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:43.309326 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-8cf2eff1-99a5-41d8-97a6-00ff95be65ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:43.317421 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 797/1557] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:43 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:43.320932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 761/1558] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:46:43 2022] POST /identity/v3/projects => generated 378 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:46:43.321088 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69634a89-6621-4f18-b53a-9123092f9202 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:43.321223 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95678000-5ff2-4f82-87b5-24466d5557f9 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:43.321302 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69634a89-6621-4f18-b53a-9123092f9202 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:43.321302 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69634a89-6621-4f18-b53a-9123092f9202 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:43.321623 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95678000-5ff2-4f82-87b5-24466d5557f9 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:43.321623 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95678000-5ff2-4f82-87b5-24466d5557f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:43.339865 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-69634a89-6621-4f18-b53a-9123092f9202 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:43.340074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-95678000-5ff2-4f82-87b5-24466d5557f9 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:43.341058 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-69634a89-6621-4f18-b53a-9123092f9202 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:43.341175 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-95678000-5ff2-4f82-87b5-24466d5557f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:43.402917 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 798/1559] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:43 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:43.407949 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 762/1560] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:43 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:43.407949 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95678000-5ff2-4f82-87b5-24466d5557f9 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:43.408151 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683d586d-e1df-4515-ae9a-b7e18b29d0c3 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:43.408151 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683d586d-e1df-4515-ae9a-b7e18b29d0c3 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:43.408151 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683d586d-e1df-4515-ae9a-b7e18b29d0c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:43.414188 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:46:43.418739 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95678000-5ff2-4f82-87b5-24466d5557f9 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:43.421565 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:46:43.424971 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db0db80-6c6f-4e04-95c9-7642a48609d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:43.425745 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db0db80-6c6f-4e04-95c9-7642a48609d9 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:43.426062 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db0db80-6c6f-4e04-95c9-7642a48609d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:43.426436 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db0db80-6c6f-4e04-95c9-7642a48609d9 admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:43.427091 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db0db80-6c6f-4e04-95c9-7642a48609d9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:46:43.427399 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db0db80-6c6f-4e04-95c9-7642a48609d9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:43.428139 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-683d586d-e1df-4515-ae9a-b7e18b29d0c3 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:43.429063 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db0db80-6c6f-4e04-95c9-7642a48609d9 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:43.429718 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-683d586d-e1df-4515-ae9a-b7e18b29d0c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:43.438017 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 799/1561] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:43 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:43.438857 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683d586d-e1df-4515-ae9a-b7e18b29d0c3 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:43.442281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 763/1562] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:46:43 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:46:43.442962 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db0db80-6c6f-4e04-95c9-7642a48609d9 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:43.449495 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683d586d-e1df-4515-ae9a-b7e18b29d0c3 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:43.454022 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121fb401-36bd-4559-86c3-6ac5dedf926c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:43.454420 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db0db80-6c6f-4e04-95c9-7642a48609d9 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:43.454680 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121fb401-36bd-4559-86c3-6ac5dedf926c admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:43.454965 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121fb401-36bd-4559-86c3-6ac5dedf926c admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:43.455239 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121fb401-36bd-4559-86c3-6ac5dedf926c admin admin] PATH_INFO: `/v3/projects/8fe57b34d5144e91bafb66f54169cbcf/users/2fe5724620da46ddbefbdfd442dc8d73/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:43.459225 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57a908a-3bb1-40cb-97e8-7361854b4a40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:43.460018 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57a908a-3bb1-40cb-97e8-7361854b4a40 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:43.460292 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57a908a-3bb1-40cb-97e8-7361854b4a40 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:43.460560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57a908a-3bb1-40cb-97e8-7361854b4a40 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:43.461870 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57a908a-3bb1-40cb-97e8-7361854b4a40 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:43.463806 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57a908a-3bb1-40cb-97e8-7361854b4a40 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:43.471129 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121fb401-36bd-4559-86c3-6ac5dedf926c admin admin] RBAC: Authorizing `identity:create_grant(project_id=8fe57b34d5144e91bafb66f54169cbcf, user_id=2fe5724620da46ddbefbdfd442dc8d73, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:43.474171 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121fb401-36bd-4559-86c3-6ac5dedf926c admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:43.486975 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 800/1563] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:46:43 2022] PUT /identity/v3/projects/8fe57b34d5144e91bafb66f54169cbcf/users/2fe5724620da46ddbefbdfd442dc8d73/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:46:43.487854 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121fb401-36bd-4559-86c3-6ac5dedf926c admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:43.495754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 764/1564] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:46:43 2022] POST /identity/v3/projects => generated 376 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:46:43.497866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57a908a-3bb1-40cb-97e8-7361854b4a40 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:43.551123 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121fb401-36bd-4559-86c3-6ac5dedf926c admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:43.562098 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57a908a-3bb1-40cb-97e8-7361854b4a40 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:43.567013 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4868339-6d94-4892-95d1-e936b5b33c16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:43.567615 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4868339-6d94-4892-95d1-e936b5b33c16 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:43.567921 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4868339-6d94-4892-95d1-e936b5b33c16 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:43.568199 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4868339-6d94-4892-95d1-e936b5b33c16 admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:43.568850 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4868339-6d94-4892-95d1-e936b5b33c16 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:46:43.569156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4868339-6d94-4892-95d1-e936b5b33c16 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:43.571287 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4868339-6d94-4892-95d1-e936b5b33c16 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:43.583109 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 765/1565] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:46:43 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:46:43.585407 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e53f56-99db-4ef3-8d16-dd261ddf53d4 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:43.585704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e53f56-99db-4ef3-8d16-dd261ddf53d4 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:43.587852 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e53f56-99db-4ef3-8d16-dd261ddf53d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:43.608695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-62e53f56-99db-4ef3-8d16-dd261ddf53d4 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:43.609673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40928a1b-e8cd-4157-b9ff-79681fbfb15e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:43.609968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-62e53f56-99db-4ef3-8d16-dd261ddf53d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:43.610438 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40928a1b-e8cd-4157-b9ff-79681fbfb15e admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:43.610784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40928a1b-e8cd-4157-b9ff-79681fbfb15e admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:43.611053 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40928a1b-e8cd-4157-b9ff-79681fbfb15e admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:43.611705 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40928a1b-e8cd-4157-b9ff-79681fbfb15e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:43.612718 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: WARNING py.warnings [None req-40928a1b-e8cd-4157-b9ff-79681fbfb15e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 24 15:46:43.612718 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: warnings.warn(msg) Aug 24 15:46:43.612718 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: Aug 24 15:46:43.613921 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40928a1b-e8cd-4157-b9ff-79681fbfb15e admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:43.618125 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 766/1566] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:43 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:43.619366 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5372452d-582a-465b-beab-cbebc5421e86 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:43.619739 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5372452d-582a-465b-beab-cbebc5421e86 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:43.620071 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5372452d-582a-465b-beab-cbebc5421e86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:43.636356 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 801/1567] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:46:43 2022] POST /identity/v3/users => generated 290 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:46:43.637185 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40928a1b-e8cd-4157-b9ff-79681fbfb15e admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:43.641393 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-5372452d-582a-465b-beab-cbebc5421e86 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:43.642625 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-5372452d-582a-465b-beab-cbebc5421e86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:43.647376 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40928a1b-e8cd-4157-b9ff-79681fbfb15e admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:43.651288 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 767/1568] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:43 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:43.652780 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5372452d-582a-465b-beab-cbebc5421e86 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:43.653100 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb75efb9-c9db-42ec-8cca-fd80c5470ce9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:43.653911 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb75efb9-c9db-42ec-8cca-fd80c5470ce9 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:43.654195 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb75efb9-c9db-42ec-8cca-fd80c5470ce9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:43.654565 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb75efb9-c9db-42ec-8cca-fd80c5470ce9 admin admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:43.655307 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb75efb9-c9db-42ec-8cca-fd80c5470ce9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:46:43.655617 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb75efb9-c9db-42ec-8cca-fd80c5470ce9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:43.657246 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb75efb9-c9db-42ec-8cca-fd80c5470ce9 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:43.662765 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5372452d-582a-465b-beab-cbebc5421e86 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:43.668576 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a979c184-263e-4039-9164-2c683ddb1894 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:43.669161 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a979c184-263e-4039-9164-2c683ddb1894 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:43.669441 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a979c184-263e-4039-9164-2c683ddb1894 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:43.669714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a979c184-263e-4039-9164-2c683ddb1894 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:43.670329 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 802/1569] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:46:43 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:46:43.670440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a979c184-263e-4039-9164-2c683ddb1894 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:43.671527 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60db88e-1acc-49da-8b5e-51138a30148b None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:43.671838 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60db88e-1acc-49da-8b5e-51138a30148b None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:43.672113 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60db88e-1acc-49da-8b5e-51138a30148b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:43.672267 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a979c184-263e-4039-9164-2c683ddb1894 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:43.691457 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-c60db88e-1acc-49da-8b5e-51138a30148b None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:43.692697 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-c60db88e-1acc-49da-8b5e-51138a30148b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:43.707769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 803/1570] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:43 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:43.707769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60db88e-1acc-49da-8b5e-51138a30148b None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:43.714967 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 768/1571] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:46:43 2022] POST /identity/v3/projects => generated 372 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:46:43.716217 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a979c184-263e-4039-9164-2c683ddb1894 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:43.725105 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60db88e-1acc-49da-8b5e-51138a30148b None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:43.778216 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:46:43.783967 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a979c184-263e-4039-9164-2c683ddb1894 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:43.787927 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c421c2f-68a2-46fe-9266-4db31ac6cda2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:43.788542 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c421c2f-68a2-46fe-9266-4db31ac6cda2 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:43.788823 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c421c2f-68a2-46fe-9266-4db31ac6cda2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:43.788939 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0158947-3edc-4fcf-8355-9e957a851717 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:43.789097 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c421c2f-68a2-46fe-9266-4db31ac6cda2 admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:43.789560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0158947-3edc-4fcf-8355-9e957a851717 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:43.789768 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c421c2f-68a2-46fe-9266-4db31ac6cda2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:43.789900 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0158947-3edc-4fcf-8355-9e957a851717 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:43.790102 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0158947-3edc-4fcf-8355-9e957a851717 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:43.790728 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0158947-3edc-4fcf-8355-9e957a851717 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:43.791721 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c421c2f-68a2-46fe-9266-4db31ac6cda2 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:43.792849 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0158947-3edc-4fcf-8355-9e957a851717 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:43.814817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 804/1572] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:46:43 2022] POST /identity/v3/users => generated 289 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:46:43.815556 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c421c2f-68a2-46fe-9266-4db31ac6cda2 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:43.822051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 769/1573] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:46:43 2022] POST /identity/v3/projects => generated 374 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:46:43.822955 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0158947-3edc-4fcf-8355-9e957a851717 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:43.831310 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c421c2f-68a2-46fe-9266-4db31ac6cda2 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:43.836887 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:46:43.887972 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0158947-3edc-4fcf-8355-9e957a851717 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:43.892568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d7d898-4be4-419b-b4a6-62984412c271 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:43.892751 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a5f233-87fb-46ba-b1fa-912bbfe89819 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:43.893192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d7d898-4be4-419b-b4a6-62984412c271 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:43.893469 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d7d898-4be4-419b-b4a6-62984412c271 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:43.893573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a5f233-87fb-46ba-b1fa-912bbfe89819 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:43.893739 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d7d898-4be4-419b-b4a6-62984412c271 admin admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:43.893836 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a5f233-87fb-46ba-b1fa-912bbfe89819 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:43.894027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a5f233-87fb-46ba-b1fa-912bbfe89819 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:43.894502 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d7d898-4be4-419b-b4a6-62984412c271 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:46:43.894676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a5f233-87fb-46ba-b1fa-912bbfe89819 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:43.894813 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d7d898-4be4-419b-b4a6-62984412c271 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:43.896340 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a5f233-87fb-46ba-b1fa-912bbfe89819 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:43.896490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d7d898-4be4-419b-b4a6-62984412c271 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:43.909415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 770/1574] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:46:43 2022] GET /identity/v3/roles => generated 1427 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:46:43.909415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a5f233-87fb-46ba-b1fa-912bbfe89819 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:43.917584 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 805/1575] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:46:43 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:46:43.917584 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d7d898-4be4-419b-b4a6-62984412c271 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:43.917907 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a5f233-87fb-46ba-b1fa-912bbfe89819 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:43.921161 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d7d898-4be4-419b-b4a6-62984412c271 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:43.922901 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a04e66-7720-4957-a526-c419dd3406b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:43.923542 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a04e66-7720-4957-a526-c419dd3406b1 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:43.923855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a04e66-7720-4957-a526-c419dd3406b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:43.924137 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a04e66-7720-4957-a526-c419dd3406b1 admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:43.924816 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a04e66-7720-4957-a526-c419dd3406b1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:46:43.925123 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a04e66-7720-4957-a526-c419dd3406b1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:43.925963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948251ae-0799-4eeb-9080-d26a8dcf290c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:43.926567 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948251ae-0799-4eeb-9080-d26a8dcf290c admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:43.926724 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a04e66-7720-4957-a526-c419dd3406b1 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:43.926867 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948251ae-0799-4eeb-9080-d26a8dcf290c admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:43.927091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948251ae-0799-4eeb-9080-d26a8dcf290c admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:43.927748 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948251ae-0799-4eeb-9080-d26a8dcf290c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:43.929642 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948251ae-0799-4eeb-9080-d26a8dcf290c admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:43.939819 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 771/1576] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:46:43 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:46:43.940631 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a04e66-7720-4957-a526-c419dd3406b1 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:43.942487 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-43a04e66-7720-4957-a526-c419dd3406b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:43.967712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 806/1577] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:46:43 2022] POST /identity/v3/projects => generated 384 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:46:43.968922 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948251ae-0799-4eeb-9080-d26a8dcf290c admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:44.002681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a04e66-7720-4957-a526-c419dd3406b1 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:44.032012 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948251ae-0799-4eeb-9080-d26a8dcf290c admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:44.036739 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e121e4c-72af-4bcf-87cb-c4286dc8bb7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:44.037351 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e121e4c-72af-4bcf-87cb-c4286dc8bb7e admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:44.037632 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e121e4c-72af-4bcf-87cb-c4286dc8bb7e admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:44.037913 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e121e4c-72af-4bcf-87cb-c4286dc8bb7e admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:44.038542 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e121e4c-72af-4bcf-87cb-c4286dc8bb7e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:44.040537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e121e4c-72af-4bcf-87cb-c4286dc8bb7e admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:44.052018 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fc9ceb-34c3-4a50-903c-40c70a96aa9c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:44.052675 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fc9ceb-34c3-4a50-903c-40c70a96aa9c service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:44.052957 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fc9ceb-34c3-4a50-903c-40c70a96aa9c service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:44.053360 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fc9ceb-34c3-4a50-903c-40c70a96aa9c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:44.063240 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 807/1578] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:46:43 2022] POST /identity/v3/users => generated 287 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:46:44.063897 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59fc9ceb-34c3-4a50-903c-40c70a96aa9c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:44.064342 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e121e4c-72af-4bcf-87cb-c4286dc8bb7e admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:44.065929 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59fc9ceb-34c3-4a50-903c-40c70a96aa9c service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:44.076714 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e121e4c-72af-4bcf-87cb-c4286dc8bb7e admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:44.077202 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 772/1579] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:46:43 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:46:44.078087 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fc9ceb-34c3-4a50-903c-40c70a96aa9c service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:44.084574 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47aaa645-7996-456c-8faa-a8410adcfd7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:44.085211 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47aaa645-7996-456c-8faa-a8410adcfd7d admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:44.085538 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47aaa645-7996-456c-8faa-a8410adcfd7d admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:44.085810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47aaa645-7996-456c-8faa-a8410adcfd7d admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:44.087676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47aaa645-7996-456c-8faa-a8410adcfd7d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:44.088740 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fc9ceb-34c3-4a50-903c-40c70a96aa9c service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:44.089476 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47aaa645-7996-456c-8faa-a8410adcfd7d admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:44.093523 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944ced41-9ce4-4dc0-8958-2876102f2b78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:44.094239 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944ced41-9ce4-4dc0-8958-2876102f2b78 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:44.094586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944ced41-9ce4-4dc0-8958-2876102f2b78 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:44.094929 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944ced41-9ce4-4dc0-8958-2876102f2b78 admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:44.095667 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-944ced41-9ce4-4dc0-8958-2876102f2b78 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:44.097662 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-944ced41-9ce4-4dc0-8958-2876102f2b78 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:44.106603 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 808/1580] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:46:44 2022] GET /identity/v3/roles => generated 1427 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:46:44.106603 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47aaa645-7996-456c-8faa-a8410adcfd7d admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:44.118784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47aaa645-7996-456c-8faa-a8410adcfd7d admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:44.123874 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 773/1581] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:46:44 2022] POST /identity/v3/users => generated 288 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:46:44.124658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944ced41-9ce4-4dc0-8958-2876102f2b78 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:44.129213 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6522d9-507a-4d52-b72d-acc091009c0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:44.129935 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6522d9-507a-4d52-b72d-acc091009c0e admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:44.130222 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6522d9-507a-4d52-b72d-acc091009c0e admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:44.130557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6522d9-507a-4d52-b72d-acc091009c0e admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:44.131221 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6522d9-507a-4d52-b72d-acc091009c0e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:44.135243 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6522d9-507a-4d52-b72d-acc091009c0e admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:44.137416 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944ced41-9ce4-4dc0-8958-2876102f2b78 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:44.142794 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072a5205-a2d6-47f6-9680-eac4f6b7fe3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:44.143406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072a5205-a2d6-47f6-9680-eac4f6b7fe3a admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:44.143709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072a5205-a2d6-47f6-9680-eac4f6b7fe3a admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:44.144047 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072a5205-a2d6-47f6-9680-eac4f6b7fe3a admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:44.144686 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-072a5205-a2d6-47f6-9680-eac4f6b7fe3a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:44.149675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 809/1582] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:46:44 2022] GET /identity/v3/roles => generated 1427 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:46:44.149832 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-072a5205-a2d6-47f6-9680-eac4f6b7fe3a admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:44.149961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6522d9-507a-4d52-b72d-acc091009c0e admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:44.160652 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6522d9-507a-4d52-b72d-acc091009c0e admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:44.167793 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04bb2c2-6d75-4607-9d07-934dbc826778 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:44.168468 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04bb2c2-6d75-4607-9d07-934dbc826778 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:44.168813 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04bb2c2-6d75-4607-9d07-934dbc826778 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:44.169130 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04bb2c2-6d75-4607-9d07-934dbc826778 admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:44.169872 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04bb2c2-6d75-4607-9d07-934dbc826778 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:44.172117 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04bb2c2-6d75-4607-9d07-934dbc826778 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:44.181271 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 774/1583] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:46:44 2022] POST /identity/v3/projects => generated 388 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:46:44.182596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072a5205-a2d6-47f6-9680-eac4f6b7fe3a admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:44.198646 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:46:44.202972 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:46:44.213489 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 810/1584] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:46:44 2022] POST /identity/v3/projects => generated 382 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:46:44.213489 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04bb2c2-6d75-4607-9d07-934dbc826778 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:44.263940 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072a5205-a2d6-47f6-9680-eac4f6b7fe3a admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:44.303460 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04bb2c2-6d75-4607-9d07-934dbc826778 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:44.309962 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f6f56d-4b65-43ba-aee1-29f2e48bd745 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:44.310728 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f6f56d-4b65-43ba-aee1-29f2e48bd745 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:44.310861 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f6f56d-4b65-43ba-aee1-29f2e48bd745 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:44.311140 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f6f56d-4b65-43ba-aee1-29f2e48bd745 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:44.311827 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f6f56d-4b65-43ba-aee1-29f2e48bd745 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:44.313582 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f6f56d-4b65-43ba-aee1-29f2e48bd745 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:44.326711 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 811/1585] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:46:44 2022] GET /identity/v3/roles => generated 1427 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:46:44.326711 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f6f56d-4b65-43ba-aee1-29f2e48bd745 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:44.337657 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f163ab84-8caa-4adb-b019-6f2e8a6e7888 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:44.338488 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f163ab84-8caa-4adb-b019-6f2e8a6e7888 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:44.338960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f163ab84-8caa-4adb-b019-6f2e8a6e7888 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:44.339237 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f163ab84-8caa-4adb-b019-6f2e8a6e7888 admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:44.339764 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f6f56d-4b65-43ba-aee1-29f2e48bd745 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:44.339970 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f163ab84-8caa-4adb-b019-6f2e8a6e7888 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:44.341904 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f163ab84-8caa-4adb-b019-6f2e8a6e7888 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:44.345227 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc27a14-6623-4107-9092-ea602a111d93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:44.346440 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc27a14-6623-4107-9092-ea602a111d93 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:44.346734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc27a14-6623-4107-9092-ea602a111d93 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:44.347018 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc27a14-6623-4107-9092-ea602a111d93 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:44.347875 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc27a14-6623-4107-9092-ea602a111d93 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:44.349542 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc27a14-6623-4107-9092-ea602a111d93 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:44.361887 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 812/1586] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:46:44 2022] GET /identity/v3/roles => generated 1427 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:46:44.362604 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc27a14-6623-4107-9092-ea602a111d93 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:44.370003 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 775/1587] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:46:44 2022] POST /identity/v3/users => generated 293 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:46:44.371145 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f163ab84-8caa-4adb-b019-6f2e8a6e7888 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:44.377191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc27a14-6623-4107-9092-ea602a111d93 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:44.382855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f163ab84-8caa-4adb-b019-6f2e8a6e7888 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:44.383776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee00e3b-e558-4460-afa0-19e0ab88c703 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:44.384392 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee00e3b-e558-4460-afa0-19e0ab88c703 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:44.384673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee00e3b-e558-4460-afa0-19e0ab88c703 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:44.384941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee00e3b-e558-4460-afa0-19e0ab88c703 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:44.385707 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee00e3b-e558-4460-afa0-19e0ab88c703 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:44.387550 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee00e3b-e558-4460-afa0-19e0ab88c703 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:44.390172 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76478e69-265f-424a-86b6-8f2a9a8d311a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:44.392131 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76478e69-265f-424a-86b6-8f2a9a8d311a admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:44.392362 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76478e69-265f-424a-86b6-8f2a9a8d311a admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:44.392829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76478e69-265f-424a-86b6-8f2a9a8d311a admin admin] PATH_INFO: `/v3/projects/b454a2613ce5453b81b0a78019eed614/users/9db03e18a7b24157a8ac724a615373f4/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:44.400999 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 813/1588] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:46:44 2022] GET /identity/v3/roles => generated 1427 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:46:44.401787 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee00e3b-e558-4460-afa0-19e0ab88c703 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:44.413135 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee00e3b-e558-4460-afa0-19e0ab88c703 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:44.414721 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76478e69-265f-424a-86b6-8f2a9a8d311a admin admin] RBAC: Authorizing `identity:create_grant(project_id=b454a2613ce5453b81b0a78019eed614, user_id=9db03e18a7b24157a8ac724a615373f4, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:44.415900 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: WARNING py.warnings [None req-76478e69-265f-424a-86b6-8f2a9a8d311a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 24 15:46:44.415900 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: warnings.warn(msg) Aug 24 15:46:44.415900 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: Aug 24 15:46:44.417932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76478e69-265f-424a-86b6-8f2a9a8d311a admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:44.419504 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3729386-37d3-4ae3-b3aa-e08a3b05c91b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:44.420517 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3729386-37d3-4ae3-b3aa-e08a3b05c91b admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:44.420866 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3729386-37d3-4ae3-b3aa-e08a3b05c91b admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:44.421215 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3729386-37d3-4ae3-b3aa-e08a3b05c91b admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:44.421910 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3729386-37d3-4ae3-b3aa-e08a3b05c91b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:44.424018 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3729386-37d3-4ae3-b3aa-e08a3b05c91b admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:44.435283 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 776/1589] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:46:44 2022] PUT /identity/v3/projects/b454a2613ce5453b81b0a78019eed614/users/9db03e18a7b24157a8ac724a615373f4/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:46:44.436740 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76478e69-265f-424a-86b6-8f2a9a8d311a admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:44.456089 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 814/1590] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:46:44 2022] POST /identity/v3/users => generated 295 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:46:44.456089 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3729386-37d3-4ae3-b3aa-e08a3b05c91b admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:44.465476 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3729386-37d3-4ae3-b3aa-e08a3b05c91b admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:44.516604 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76478e69-265f-424a-86b6-8f2a9a8d311a admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:44.521941 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4714e38b-4a17-4ed4-ba21-b8d005a04fd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:44.522699 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4714e38b-4a17-4ed4-ba21-b8d005a04fd7 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:44.523052 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4714e38b-4a17-4ed4-ba21-b8d005a04fd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:44.523395 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4714e38b-4a17-4ed4-ba21-b8d005a04fd7 admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:44.524658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4714e38b-4a17-4ed4-ba21-b8d005a04fd7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:44.526668 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4714e38b-4a17-4ed4-ba21-b8d005a04fd7 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:44.550290 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e131d97-71c9-42ce-85d8-535ea2e0e479 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:44.551037 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e131d97-71c9-42ce-85d8-535ea2e0e479 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:44.551426 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e131d97-71c9-42ce-85d8-535ea2e0e479 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:44.552010 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e131d97-71c9-42ce-85d8-535ea2e0e479 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:44.552517 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e131d97-71c9-42ce-85d8-535ea2e0e479 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:44.554472 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e131d97-71c9-42ce-85d8-535ea2e0e479 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:44.556949 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 777/1591] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:46:44 2022] POST /identity/v3/users => generated 292 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:46:44.558073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4714e38b-4a17-4ed4-ba21-b8d005a04fd7 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:44.568426 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 815/1592] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:46:44 2022] GET /identity/v3/roles => generated 1427 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:46:44.569477 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e131d97-71c9-42ce-85d8-535ea2e0e479 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:44.575769 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4714e38b-4a17-4ed4-ba21-b8d005a04fd7 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:44.581332 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e131d97-71c9-42ce-85d8-535ea2e0e479 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:44.581557 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596a9e63-ae6d-4d77-a871-61bc6dfbc750 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:44.582300 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596a9e63-ae6d-4d77-a871-61bc6dfbc750 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:44.583101 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596a9e63-ae6d-4d77-a871-61bc6dfbc750 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:44.583452 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596a9e63-ae6d-4d77-a871-61bc6dfbc750 admin admin] PATH_INFO: `/v3/projects/b3568d52e7dd45b58640c06eddc5ea44/users/38f7d31cf0f541f58af7d3d850af6472/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:44.587282 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3057ee41-c648-4ee8-8e94-9f6c8dc0e6cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:44.588908 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3057ee41-c648-4ee8-8e94-9f6c8dc0e6cf admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:44.589285 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3057ee41-c648-4ee8-8e94-9f6c8dc0e6cf admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:44.589797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3057ee41-c648-4ee8-8e94-9f6c8dc0e6cf admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:44.590530 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3057ee41-c648-4ee8-8e94-9f6c8dc0e6cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:44.592238 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3057ee41-c648-4ee8-8e94-9f6c8dc0e6cf admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:44.602821 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-596a9e63-ae6d-4d77-a871-61bc6dfbc750 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b3568d52e7dd45b58640c06eddc5ea44, user_id=38f7d31cf0f541f58af7d3d850af6472, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:44.605607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-596a9e63-ae6d-4d77-a871-61bc6dfbc750 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:44.606974 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 816/1593] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:46:44 2022] GET /identity/v3/roles => generated 1427 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:46:44.607829 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3057ee41-c648-4ee8-8e94-9f6c8dc0e6cf admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:44.608664 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:46:44.618606 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3057ee41-c648-4ee8-8e94-9f6c8dc0e6cf admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:44.625090 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 778/1594] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:46:44 2022] PUT /identity/v3/projects/b3568d52e7dd45b58640c06eddc5ea44/users/38f7d31cf0f541f58af7d3d850af6472/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:46:44.629929 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52e1fb9-860b-4f21-a1dc-7683d70cd648 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:44.630166 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52e1fb9-860b-4f21-a1dc-7683d70cd648 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:44.630551 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52e1fb9-860b-4f21-a1dc-7683d70cd648 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:44.653665 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-c52e1fb9-860b-4f21-a1dc-7683d70cd648 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:44.655070 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-c52e1fb9-860b-4f21-a1dc-7683d70cd648 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:44.669388 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:46:44.695411 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97b365b-4249-4476-9464-def4c83b3436 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:44.697668 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97b365b-4249-4476-9464-def4c83b3436 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:44.698058 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97b365b-4249-4476-9464-def4c83b3436 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:44.698486 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97b365b-4249-4476-9464-def4c83b3436 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:44.699184 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97b365b-4249-4476-9464-def4c83b3436 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:44.701841 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97b365b-4249-4476-9464-def4c83b3436 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:44.715439 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 817/1595] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:46:44 2022] GET /identity/v3/roles => generated 1427 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:46:44.716527 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97b365b-4249-4476-9464-def4c83b3436 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:44.727071 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 779/1596] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:44 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:44.727518 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97b365b-4249-4476-9464-def4c83b3436 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:44.731198 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52e1fb9-860b-4f21-a1dc-7683d70cd648 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:44.733418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9cb6737-02cd-4c04-85c6-2742c25e29f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:44.734229 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cb6737-02cd-4c04-85c6-2742c25e29f7 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:44.734536 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cb6737-02cd-4c04-85c6-2742c25e29f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:44.734910 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cb6737-02cd-4c04-85c6-2742c25e29f7 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:44.735700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9cb6737-02cd-4c04-85c6-2742c25e29f7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:44.737797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9cb6737-02cd-4c04-85c6-2742c25e29f7 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:44.744762 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52e1fb9-860b-4f21-a1dc-7683d70cd648 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:44.749744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 818/1597] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:46:44 2022] GET /identity/v3/roles => generated 1427 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:46:44.750003 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4309016a-8f60-4c2f-8764-828015cd4035 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:44.750687 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4309016a-8f60-4c2f-8764-828015cd4035 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:44.750819 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9cb6737-02cd-4c04-85c6-2742c25e29f7 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:44.751094 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4309016a-8f60-4c2f-8764-828015cd4035 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:44.751434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4309016a-8f60-4c2f-8764-828015cd4035 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:44.752174 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4309016a-8f60-4c2f-8764-828015cd4035 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:44.753966 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4309016a-8f60-4c2f-8764-828015cd4035 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:44.763211 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9cb6737-02cd-4c04-85c6-2742c25e29f7 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:44.768983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 780/1598] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:46:44 2022] GET /identity/v3/roles => generated 1427 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:46:44.768983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4309016a-8f60-4c2f-8764-828015cd4035 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:44.772035 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991cfa40-446f-4d03-8e04-fbd414900098 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:44.772961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991cfa40-446f-4d03-8e04-fbd414900098 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:44.773450 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991cfa40-446f-4d03-8e04-fbd414900098 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:44.773758 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991cfa40-446f-4d03-8e04-fbd414900098 admin admin] PATH_INFO: `/v3/projects/e64b544bd42945e8bc11abeb9cf4cf0a/users/2f3ce0702d194ad9a78aed3a4fefe205/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:44.778573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4309016a-8f60-4c2f-8764-828015cd4035 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:44.784773 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af868348-a393-401b-afc6-eeec95cac094 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:44.785454 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af868348-a393-401b-afc6-eeec95cac094 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:44.785827 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af868348-a393-401b-afc6-eeec95cac094 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:44.786130 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af868348-a393-401b-afc6-eeec95cac094 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:44.786843 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af868348-a393-401b-afc6-eeec95cac094 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:44.788552 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af868348-a393-401b-afc6-eeec95cac094 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:44.791114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-991cfa40-446f-4d03-8e04-fbd414900098 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e64b544bd42945e8bc11abeb9cf4cf0a, user_id=2f3ce0702d194ad9a78aed3a4fefe205, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:44.794493 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-991cfa40-446f-4d03-8e04-fbd414900098 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:44.799339 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 781/1599] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:46:44 2022] GET /identity/v3/roles => generated 1427 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:46:44.800344 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691d0d17-dd75-4db4-aa8b-6ebf692a348c None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:44.800585 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691d0d17-dd75-4db4-aa8b-6ebf692a348c None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:44.800865 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691d0d17-dd75-4db4-aa8b-6ebf692a348c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:44.815955 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 819/1600] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:46:44 2022] PUT /identity/v3/projects/e64b544bd42945e8bc11abeb9cf4cf0a/users/2f3ce0702d194ad9a78aed3a4fefe205/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:46:44.817624 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991cfa40-446f-4d03-8e04-fbd414900098 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:44.833766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-691d0d17-dd75-4db4-aa8b-6ebf692a348c None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:44.835401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-691d0d17-dd75-4db4-aa8b-6ebf692a348c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:44.921722 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991cfa40-446f-4d03-8e04-fbd414900098 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:44.933738 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66a66ea-c394-459b-9510-6e361a17461f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:44.934827 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66a66ea-c394-459b-9510-6e361a17461f admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:44.935349 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66a66ea-c394-459b-9510-6e361a17461f admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:44.935891 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66a66ea-c394-459b-9510-6e361a17461f admin admin] PATH_INFO: `/v3/projects/1e05694ccd044a8a88ce024ac2d0668d/users/ec9f1fb05ef1400abf74f1926d2ec385/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:44.938165 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 782/1601] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:44 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:44.944140 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691d0d17-dd75-4db4-aa8b-6ebf692a348c None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:44.965478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66a66ea-c394-459b-9510-6e361a17461f admin admin] RBAC: Authorizing `identity:create_grant(project_id=1e05694ccd044a8a88ce024ac2d0668d, user_id=ec9f1fb05ef1400abf74f1926d2ec385, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:44.969020 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66a66ea-c394-459b-9510-6e361a17461f admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:44.999650 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 820/1602] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:46:44 2022] PUT /identity/v3/projects/1e05694ccd044a8a88ce024ac2d0668d/users/ec9f1fb05ef1400abf74f1926d2ec385/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:46:44.999650 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66a66ea-c394-459b-9510-6e361a17461f admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:45.036485 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691d0d17-dd75-4db4-aa8b-6ebf692a348c None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:45.075065 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:46:45.105888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66a66ea-c394-459b-9510-6e361a17461f admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:45.111364 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6d0840-437b-4020-b7ba-798c1dac32a9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:45.112039 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6d0840-437b-4020-b7ba-798c1dac32a9 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:45.112324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6d0840-437b-4020-b7ba-798c1dac32a9 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:45.112580 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68bdfea-02db-4bb2-8931-35e50f0238a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:45.112701 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6d0840-437b-4020-b7ba-798c1dac32a9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:45.113353 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68bdfea-02db-4bb2-8931-35e50f0238a6 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:45.113481 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68bdfea-02db-4bb2-8931-35e50f0238a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:45.114577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68bdfea-02db-4bb2-8931-35e50f0238a6 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:45.114577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e68bdfea-02db-4bb2-8931-35e50f0238a6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:45.116492 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e68bdfea-02db-4bb2-8931-35e50f0238a6 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:45.125603 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f6d0840-437b-4020-b7ba-798c1dac32a9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:45.127311 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f6d0840-437b-4020-b7ba-798c1dac32a9 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:45.135682 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 821/1603] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:46:44 2022] GET /identity/v3/roles => generated 1427 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:46:45.137413 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68bdfea-02db-4bb2-8931-35e50f0238a6 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:45.141593 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 783/1604] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:46:44 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 203 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:46:45.142542 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6d0840-437b-4020-b7ba-798c1dac32a9 service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:45.150329 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68bdfea-02db-4bb2-8931-35e50f0238a6 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:45.157638 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49acc24-d9cb-4286-8c36-e9a34abb70ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:45.157888 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6d0840-437b-4020-b7ba-798c1dac32a9 service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:45.158440 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49acc24-d9cb-4286-8c36-e9a34abb70ad admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:45.158797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49acc24-d9cb-4286-8c36-e9a34abb70ad admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:45.159522 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49acc24-d9cb-4286-8c36-e9a34abb70ad admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:45.159894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b49acc24-d9cb-4286-8c36-e9a34abb70ad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:45.163173 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b49acc24-d9cb-4286-8c36-e9a34abb70ad admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:45.173317 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2798cbd7-7b3d-4309-9eb9-93e24d8de6f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:45.176145 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2798cbd7-7b3d-4309-9eb9-93e24d8de6f4 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:45.177148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2798cbd7-7b3d-4309-9eb9-93e24d8de6f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:45.177503 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2798cbd7-7b3d-4309-9eb9-93e24d8de6f4 admin admin] PATH_INFO: `/v3/projects/42da093227fd487c8df40a14d4a38c1a/users/b7052b97e46e449685a65537f3b11864/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:45.195985 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 822/1605] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:46:45 2022] GET /identity/v3/roles => generated 1427 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:46:45.197528 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3ad47c-1971-4d68-9597-a8d0b6bf7ec4 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:45.197857 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3ad47c-1971-4d68-9597-a8d0b6bf7ec4 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:45.198117 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3ad47c-1971-4d68-9597-a8d0b6bf7ec4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:45.217985 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2798cbd7-7b3d-4309-9eb9-93e24d8de6f4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=42da093227fd487c8df40a14d4a38c1a, user_id=b7052b97e46e449685a65537f3b11864, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:45.221573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2798cbd7-7b3d-4309-9eb9-93e24d8de6f4 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:45.245767 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:46:45.246968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 784/1606] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:46:45 2022] PUT /identity/v3/projects/42da093227fd487c8df40a14d4a38c1a/users/b7052b97e46e449685a65537f3b11864/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:46:45.249910 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2798cbd7-7b3d-4309-9eb9-93e24d8de6f4 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:45.258977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-8d3ad47c-1971-4d68-9597-a8d0b6bf7ec4 None None] MFA Rules not processed for user `2f3ce0702d194ad9a78aed3a4fefe205`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:45.267955 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-8d3ad47c-1971-4d68-9597-a8d0b6bf7ec4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:45.340049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2798cbd7-7b3d-4309-9eb9-93e24d8de6f4 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:45.347399 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1edaa7-6d85-4641-b0a3-b7a09e902621 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:45.350753 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1edaa7-6d85-4641-b0a3-b7a09e902621 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:45.350753 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1edaa7-6d85-4641-b0a3-b7a09e902621 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:45.350753 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1edaa7-6d85-4641-b0a3-b7a09e902621 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:45.428388 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 823/1607] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:45 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:45.428609 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af1edaa7-6d85-4641-b0a3-b7a09e902621 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:45.430545 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af1edaa7-6d85-4641-b0a3-b7a09e902621 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:45.430805 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538c2e16-ab1a-4583-9e5f-793722d346d1 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:45.431337 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538c2e16-ab1a-4583-9e5f-793722d346d1 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:45.431739 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538c2e16-ab1a-4583-9e5f-793722d346d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:45.442061 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 785/1608] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:46:45 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:46:45.442888 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1edaa7-6d85-4641-b0a3-b7a09e902621 service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:45.453593 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1edaa7-6d85-4641-b0a3-b7a09e902621 service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:45.459958 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c34173-41b6-4f6f-b2f1-56331a8421cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:45.460791 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c34173-41b6-4f6f-b2f1-56331a8421cb admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:45.461798 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c34173-41b6-4f6f-b2f1-56331a8421cb admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:45.462197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c34173-41b6-4f6f-b2f1-56331a8421cb admin admin] PATH_INFO: `/v3/projects/0b5c7713b8574a09b7af9ab97b5ecbe1/users/c058b07c1e9e443ebacbf38b1f13e58b/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:45.480114 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c34173-41b6-4f6f-b2f1-56331a8421cb admin admin] RBAC: Authorizing `identity:create_grant(project_id=0b5c7713b8574a09b7af9ab97b5ecbe1, user_id=c058b07c1e9e443ebacbf38b1f13e58b, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:45.483767 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c34173-41b6-4f6f-b2f1-56331a8421cb admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:45.500070 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-538c2e16-ab1a-4583-9e5f-793722d346d1 None None] MFA Rules not processed for user `ec9f1fb05ef1400abf74f1926d2ec385`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:45.501386 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-538c2e16-ab1a-4583-9e5f-793722d346d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:45.502766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 786/1609] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:46:45 2022] PUT /identity/v3/projects/0b5c7713b8574a09b7af9ab97b5ecbe1/users/c058b07c1e9e443ebacbf38b1f13e58b/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:46:45.504229 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c34173-41b6-4f6f-b2f1-56331a8421cb admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:45.560064 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:46:45.608774 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c34173-41b6-4f6f-b2f1-56331a8421cb admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:45.625391 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f52d21-9782-4bc5-bc4b-9df219308874 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:45.627176 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f52d21-9782-4bc5-bc4b-9df219308874 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:45.627462 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f52d21-9782-4bc5-bc4b-9df219308874 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:45.627764 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f52d21-9782-4bc5-bc4b-9df219308874 admin admin] PATH_INFO: `/v3/projects/82ee4cb082d14ccfba49cfa0ef25394e/users/29e13bb3a9584a0e8336224579fc2bfe/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:45.663544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f52d21-9782-4bc5-bc4b-9df219308874 admin admin] RBAC: Authorizing `identity:create_grant(project_id=82ee4cb082d14ccfba49cfa0ef25394e, user_id=29e13bb3a9584a0e8336224579fc2bfe, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:45.673213 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f52d21-9782-4bc5-bc4b-9df219308874 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:45.689673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 824/1610] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:45 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 257 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:45.689673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbf8282-eba5-4815-a0e5-f569bd437e8a None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:45.689673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbf8282-eba5-4815-a0e5-f569bd437e8a None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:45.689673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbf8282-eba5-4815-a0e5-f569bd437e8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:45.704398 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 787/1611] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:46:45 2022] PUT /identity/v3/projects/82ee4cb082d14ccfba49cfa0ef25394e/users/29e13bb3a9584a0e8336224579fc2bfe/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:46:45.719809 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e5806b-c80a-4e4a-8e46-a71a0b488d31 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:45.720432 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e5806b-c80a-4e4a-8e46-a71a0b488d31 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:45.721877 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e5806b-c80a-4e4a-8e46-a71a0b488d31 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:45.725452 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 788/1612] 10.222.0.102 () {54 vars in 967 bytes} [Wed Aug 24 15:46:45 2022] GET /identity => generated 271 bytes in 21 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:46:45.728297 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2a0618-fdb0-412a-986c-de57c1387815 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:45.729564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2a0618-fdb0-412a-986c-de57c1387815 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:45.730032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2a0618-fdb0-412a-986c-de57c1387815 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:45.744553 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-fcbf8282-eba5-4815-a0e5-f569bd437e8a None None] MFA Rules not processed for user `b7052b97e46e449685a65537f3b11864`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:45.745925 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-fcbf8282-eba5-4815-a0e5-f569bd437e8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:45.773116 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-3f2a0618-fdb0-412a-986c-de57c1387815 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:45.776732 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-3f2a0618-fdb0-412a-986c-de57c1387815 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:45.874563 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 825/1613] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:45 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:45.875943 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbf8282-eba5-4815-a0e5-f569bd437e8a None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:45.917209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:46:45.925956 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 789/1614] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:45 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:45.927271 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e8d53b-ef26-4481-bb3f-b8cf55d63350 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:45.927561 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e8d53b-ef26-4481-bb3f-b8cf55d63350 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:45.927895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e8d53b-ef26-4481-bb3f-b8cf55d63350 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:45.990766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-11e8d53b-ef26-4481-bb3f-b8cf55d63350 None None] MFA Rules not processed for user `29e13bb3a9584a0e8336224579fc2bfe`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:45.993979 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-11e8d53b-ef26-4481-bb3f-b8cf55d63350 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:46.032607 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbf8282-eba5-4815-a0e5-f569bd437e8a None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:46.038696 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18819cc2-bee4-462f-89ec-627692e7bfeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:46.040534 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18819cc2-bee4-462f-89ec-627692e7bfeb admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:46.041299 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18819cc2-bee4-462f-89ec-627692e7bfeb admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:46.041666 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18819cc2-bee4-462f-89ec-627692e7bfeb admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:46.042428 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18819cc2-bee4-462f-89ec-627692e7bfeb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:46.044607 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18819cc2-bee4-462f-89ec-627692e7bfeb admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:46.088412 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 826/1615] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:46:45 2022] POST /identity/v3/projects => generated 372 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:46:46.090373 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7e09fe-008c-4761-8946-ee44a03e9b3a None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:46.090899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7e09fe-008c-4761-8946-ee44a03e9b3a None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:46.091328 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7e09fe-008c-4761-8946-ee44a03e9b3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:46.126178 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 790/1616] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:45 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:46.126178 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af11ac7b-41a4-4e77-b1d9-8df174aa638b None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:46.126178 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af11ac7b-41a4-4e77-b1d9-8df174aa638b None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:46.126178 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af11ac7b-41a4-4e77-b1d9-8df174aa638b None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:46.127049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 791/1617] 10.222.0.102 () {54 vars in 967 bytes} [Wed Aug 24 15:46:46 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:46:46.127908 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af11ac7b-41a4-4e77-b1d9-8df174aa638b None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:46.203976 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-2b7e09fe-008c-4761-8946-ee44a03e9b3a None None] MFA Rules not processed for user `e0d544379fe44438ad10289b45124374`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:46.207566 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-2b7e09fe-008c-4761-8946-ee44a03e9b3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:46.343697 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:46:46.343697 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af11ac7b-41a4-4e77-b1d9-8df174aa638b None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:46.343697 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1738fb-a59e-4e9f-a766-a3ac07767013 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:46.343697 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1738fb-a59e-4e9f-a766-a3ac07767013 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:46.344465 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1738fb-a59e-4e9f-a766-a3ac07767013 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:46.344465 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1738fb-a59e-4e9f-a766-a3ac07767013 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:46.416439 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 827/1618] 10.222.0.102 () {58 vars in 1064 bytes} [Wed Aug 24 15:46:46 2022] POST /identity/v3/auth/tokens => generated 3382 bytes in 327 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:46.417337 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7e09fe-008c-4761-8946-ee44a03e9b3a None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:46.449713 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c1738fb-a59e-4e9f-a766-a3ac07767013 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:46.455509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c1738fb-a59e-4e9f-a766-a3ac07767013 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:46.486833 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 792/1619] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:46:46 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:46:46.487708 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1738fb-a59e-4e9f-a766-a3ac07767013 service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:46.507560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1738fb-a59e-4e9f-a766-a3ac07767013 service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:46.523825 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382774f7-fb9a-467d-8022-3f89c4d2c4e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:46.524495 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382774f7-fb9a-467d-8022-3f89c4d2c4e8 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:46.524782 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382774f7-fb9a-467d-8022-3f89c4d2c4e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:46.525039 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382774f7-fb9a-467d-8022-3f89c4d2c4e8 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:46.526370 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-382774f7-fb9a-467d-8022-3f89c4d2c4e8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:46.528397 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-382774f7-fb9a-467d-8022-3f89c4d2c4e8 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:46.580532 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7e09fe-008c-4761-8946-ee44a03e9b3a None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:46.594992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 793/1620] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:46:46 2022] POST /identity/v3/projects => generated 380 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:46:46.598691 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2f14ef-8c78-46c4-8e35-ddb4c9213a7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:46.598691 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2f14ef-8c78-46c4-8e35-ddb4c9213a7a admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:46.598691 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2f14ef-8c78-46c4-8e35-ddb4c9213a7a admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:46.598691 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2f14ef-8c78-46c4-8e35-ddb4c9213a7a admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:46.598996 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cec9be-2767-40a7-a600-de547ae4739e None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:46.598996 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cec9be-2767-40a7-a600-de547ae4739e None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:46.599353 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cec9be-2767-40a7-a600-de547ae4739e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:46.601105 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea2f14ef-8c78-46c4-8e35-ddb4c9213a7a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:46.605862 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea2f14ef-8c78-46c4-8e35-ddb4c9213a7a admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:46.653965 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-25cec9be-2767-40a7-a600-de547ae4739e None None] MFA Rules not processed for user `e0d544379fe44438ad10289b45124374`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:46.655479 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 828/1621] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:46:46 2022] POST /identity/v3/users => generated 293 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:46:46.659669 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-25cec9be-2767-40a7-a600-de547ae4739e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:46.661539 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2f14ef-8c78-46c4-8e35-ddb4c9213a7a admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:46.683912 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2f14ef-8c78-46c4-8e35-ddb4c9213a7a admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:46.698781 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:46:46.820294 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 794/1622] 10.222.0.102 () {58 vars in 1064 bytes} [Wed Aug 24 15:46:46 2022] POST /identity/v3/auth/tokens => generated 3382 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:46.826662 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25cec9be-2767-40a7-a600-de547ae4739e None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:46.834114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93124746-5c3e-4951-9468-a594487eb7db service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:46.835508 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93124746-5c3e-4951-9468-a594487eb7db service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:46.836352 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93124746-5c3e-4951-9468-a594487eb7db service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:46.838246 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93124746-5c3e-4951-9468-a594487eb7db service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:46.926480 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25cec9be-2767-40a7-a600-de547ae4739e None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:46.933881 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e678c2a1-6b1d-4401-ada4-df77afb198d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:46.934647 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e678c2a1-6b1d-4401-ada4-df77afb198d4 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:46.937227 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e678c2a1-6b1d-4401-ada4-df77afb198d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:46.937793 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e678c2a1-6b1d-4401-ada4-df77afb198d4 admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:46.938669 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e678c2a1-6b1d-4401-ada4-df77afb198d4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:46.942563 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e678c2a1-6b1d-4401-ada4-df77afb198d4 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:46.979775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:46:47.000853 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93124746-5c3e-4951-9468-a594487eb7db service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:47.001535 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 795/1623] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:46:46 2022] POST /identity/v3/users => generated 297 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:46:47.003269 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e678c2a1-6b1d-4401-ada4-df77afb198d4 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:47.006781 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93124746-5c3e-4951-9468-a594487eb7db service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:47.020764 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e678c2a1-6b1d-4401-ada4-df77afb198d4 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:47.029985 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8ea204-dba9-486f-b25c-718ffb47d786 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:47.031543 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8ea204-dba9-486f-b25c-718ffb47d786 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:47.032085 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8ea204-dba9-486f-b25c-718ffb47d786 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:47.032754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8ea204-dba9-486f-b25c-718ffb47d786 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:47.034254 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8ea204-dba9-486f-b25c-718ffb47d786 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:47.036466 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8ea204-dba9-486f-b25c-718ffb47d786 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:47.047979 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 829/1624] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:46:46 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:46:47.048971 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93124746-5c3e-4951-9468-a594487eb7db service nova] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:47.050812 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 796/1625] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:46:47 2022] GET /identity/v3/roles => generated 1427 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:46:47.051941 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8ea204-dba9-486f-b25c-718ffb47d786 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:47.068762 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8ea204-dba9-486f-b25c-718ffb47d786 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:47.074855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7327608-0caa-44a0-920d-84f26ab63684 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:47.076262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7327608-0caa-44a0-920d-84f26ab63684 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:47.078332 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7327608-0caa-44a0-920d-84f26ab63684 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:47.078962 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7327608-0caa-44a0-920d-84f26ab63684 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:47.081251 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7327608-0caa-44a0-920d-84f26ab63684 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:47.083995 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7327608-0caa-44a0-920d-84f26ab63684 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:47.096734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93124746-5c3e-4951-9468-a594487eb7db service nova] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:47.100705 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 797/1626] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:46:47 2022] GET /identity/v3/roles => generated 1427 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:46:47.102637 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7327608-0caa-44a0-920d-84f26ab63684 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:47.127938 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7327608-0caa-44a0-920d-84f26ab63684 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:47.129663 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d0dcc3-c163-4f78-ab75-12026ee59aa2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:47.130554 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d0dcc3-c163-4f78-ab75-12026ee59aa2 service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:47.130852 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d0dcc3-c163-4f78-ab75-12026ee59aa2 service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:47.131120 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d0dcc3-c163-4f78-ab75-12026ee59aa2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:47.143891 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13833b71-74ec-411c-bf0c-402872e3680b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:47.145290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13833b71-74ec-411c-bf0c-402872e3680b admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:47.147785 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13833b71-74ec-411c-bf0c-402872e3680b admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:47.148196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13833b71-74ec-411c-bf0c-402872e3680b admin admin] PATH_INFO: `/v3/projects/708a7189013e4dfd80f0d51e04c8156c/users/71acccdbd5de4307b949471c4dd734a7/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:47.199833 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13833b71-74ec-411c-bf0c-402872e3680b admin admin] RBAC: Authorizing `identity:create_grant(project_id=708a7189013e4dfd80f0d51e04c8156c, user_id=71acccdbd5de4307b949471c4dd734a7, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:47.208491 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13833b71-74ec-411c-bf0c-402872e3680b admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:47.238986 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 798/1627] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:46:47 2022] PUT /identity/v3/projects/708a7189013e4dfd80f0d51e04c8156c/users/71acccdbd5de4307b949471c4dd734a7/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:46:47.240149 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13833b71-74ec-411c-bf0c-402872e3680b admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:47.273295 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5d0dcc3-c163-4f78-ab75-12026ee59aa2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:47.277939 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5d0dcc3-c163-4f78-ab75-12026ee59aa2 service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:47.301843 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 830/1628] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:46:47 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 253 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:46:47.302847 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d0dcc3-c163-4f78-ab75-12026ee59aa2 service nova] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:47.327773 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d0dcc3-c163-4f78-ab75-12026ee59aa2 service nova] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:47.391685 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13833b71-74ec-411c-bf0c-402872e3680b admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:47.407858 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ce9fa8-7710-4fbe-b43f-5fef6f6a6ed7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:47.409665 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ce9fa8-7710-4fbe-b43f-5fef6f6a6ed7 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:47.412491 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ce9fa8-7710-4fbe-b43f-5fef6f6a6ed7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:47.413472 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ce9fa8-7710-4fbe-b43f-5fef6f6a6ed7 admin admin] PATH_INFO: `/v3/projects/d49879196fe94da7be642cdf7b0c4d35/users/9c5d4d66a1fa41ee9ae3b34f36ea594d/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:47.437211 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:46:47.466275 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ce9fa8-7710-4fbe-b43f-5fef6f6a6ed7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d49879196fe94da7be642cdf7b0c4d35, user_id=9c5d4d66a1fa41ee9ae3b34f36ea594d, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:47.470872 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222d1c2f-768b-40ff-a467-3af0a773f3c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:47.471519 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222d1c2f-768b-40ff-a467-3af0a773f3c0 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:47.471831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222d1c2f-768b-40ff-a467-3af0a773f3c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:47.472107 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222d1c2f-768b-40ff-a467-3af0a773f3c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:47.472766 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-222d1c2f-768b-40ff-a467-3af0a773f3c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:47.472974 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ce9fa8-7710-4fbe-b43f-5fef6f6a6ed7 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:47.474906 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-222d1c2f-768b-40ff-a467-3af0a773f3c0 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:47.504470 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 831/1629] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:46:47 2022] GET /identity/v3/roles => generated 1427 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:46:47.505406 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222d1c2f-768b-40ff-a467-3af0a773f3c0 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:47.510907 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 799/1630] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:46:47 2022] PUT /identity/v3/projects/d49879196fe94da7be642cdf7b0c4d35/users/9c5d4d66a1fa41ee9ae3b34f36ea594d/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:46:47.516532 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ce9fa8-7710-4fbe-b43f-5fef6f6a6ed7 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:47.537429 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222d1c2f-768b-40ff-a467-3af0a773f3c0 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:47.674700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84416c4-c967-48df-87d8-6615b9c269b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:47.675387 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84416c4-c967-48df-87d8-6615b9c269b5 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:47.675694 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84416c4-c967-48df-87d8-6615b9c269b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:47.675979 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84416c4-c967-48df-87d8-6615b9c269b5 admin admin] PATH_INFO: `/v3/projects/708a7189013e4dfd80f0d51e04c8156c/users/71acccdbd5de4307b949471c4dd734a7/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:47.680564 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84416c4-c967-48df-87d8-6615b9c269b5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=708a7189013e4dfd80f0d51e04c8156c, user_id=71acccdbd5de4307b949471c4dd734a7, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:47.683416 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84416c4-c967-48df-87d8-6615b9c269b5 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:47.712093 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 832/1631] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:46:47 2022] PUT /identity/v3/projects/708a7189013e4dfd80f0d51e04c8156c/users/71acccdbd5de4307b949471c4dd734a7/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:46:47.723260 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5c1139-30a3-44da-b004-bbe54322b629 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:47.723623 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5c1139-30a3-44da-b004-bbe54322b629 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:47.723864 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5c1139-30a3-44da-b004-bbe54322b629 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:47.740062 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ce9fa8-7710-4fbe-b43f-5fef6f6a6ed7 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:47.823445 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:46:47.829934 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-bd5c1139-30a3-44da-b004-bbe54322b629 None None] MFA Rules not processed for user `71acccdbd5de4307b949471c4dd734a7`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:47.831195 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-bd5c1139-30a3-44da-b004-bbe54322b629 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:47.876436 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1c9ebe-2440-4cdd-8eb5-c0f5c2b23fbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:47.879536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1c9ebe-2440-4cdd-8eb5-c0f5c2b23fbd admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:47.881083 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1c9ebe-2440-4cdd-8eb5-c0f5c2b23fbd admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:47.881473 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1c9ebe-2440-4cdd-8eb5-c0f5c2b23fbd admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:47.884319 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1c9ebe-2440-4cdd-8eb5-c0f5c2b23fbd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:47.888450 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1c9ebe-2440-4cdd-8eb5-c0f5c2b23fbd admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:47.913400 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 800/1632] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:46:47 2022] GET /identity/v3/roles => generated 1427 bytes in 398 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:46:47.918437 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43cba2f-6e97-4707-b545-13bcc0981229 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:47.918973 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43cba2f-6e97-4707-b545-13bcc0981229 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:47.919442 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43cba2f-6e97-4707-b545-13bcc0981229 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:47.923531 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 801/1633] 10.222.0.102 () {54 vars in 917 bytes} [Wed Aug 24 15:46:47 2022] GET /identity => generated 271 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:46:47.927904 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd1530f-bf72-4077-a382-1dca5ac11013 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:47.928327 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd1530f-bf72-4077-a382-1dca5ac11013 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:47.928771 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd1530f-bf72-4077-a382-1dca5ac11013 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:47.933599 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 802/1634] 10.222.0.102 () {54 vars in 917 bytes} [Wed Aug 24 15:46:47 2022] GET /identity => generated 271 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:46:47.934517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd1530f-bf72-4077-a382-1dca5ac11013 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:47.967688 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd1530f-bf72-4077-a382-1dca5ac11013 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:47.977931 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff980945-9775-48c1-af76-4c3338999119 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:47.978397 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 833/1635] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:47 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 257 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:47.981013 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d85bb0d-618d-48d0-9560-f1780cf01bea None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:47.981333 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d85bb0d-618d-48d0-9560-f1780cf01bea None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:47.981619 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d85bb0d-618d-48d0-9560-f1780cf01bea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:47.983420 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff980945-9775-48c1-af76-4c3338999119 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:47.983793 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff980945-9775-48c1-af76-4c3338999119 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:47.985103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff980945-9775-48c1-af76-4c3338999119 admin admin] PATH_INFO: `/v3/projects/d49879196fe94da7be642cdf7b0c4d35/users/9c5d4d66a1fa41ee9ae3b34f36ea594d/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:47.990949 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff980945-9775-48c1-af76-4c3338999119 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d49879196fe94da7be642cdf7b0c4d35, user_id=9c5d4d66a1fa41ee9ae3b34f36ea594d, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:47.998328 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff980945-9775-48c1-af76-4c3338999119 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:48.014642 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-6d85bb0d-618d-48d0-9560-f1780cf01bea None None] MFA Rules not processed for user `e0d544379fe44438ad10289b45124374`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:48.016457 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-6d85bb0d-618d-48d0-9560-f1780cf01bea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:48.030780 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 803/1636] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:46:47 2022] PUT /identity/v3/projects/d49879196fe94da7be642cdf7b0c4d35/users/9c5d4d66a1fa41ee9ae3b34f36ea594d/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:46:48.032258 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84aefaf2-86a4-4942-8023-1ab3ea988c52 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:48.032612 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84aefaf2-86a4-4942-8023-1ab3ea988c52 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:48.032963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84aefaf2-86a4-4942-8023-1ab3ea988c52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:48.078975 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-84aefaf2-86a4-4942-8023-1ab3ea988c52 None None] MFA Rules not processed for user `e0d544379fe44438ad10289b45124374`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:48.080269 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-84aefaf2-86a4-4942-8023-1ab3ea988c52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:48.146339 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:46:48.169219 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 834/1637] 10.222.0.102 () {58 vars in 1014 bytes} [Wed Aug 24 15:46:47 2022] POST /identity/v3/auth/tokens => generated 3382 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:48.171298 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d85bb0d-618d-48d0-9560-f1780cf01bea None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:48.236489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 804/1638] 10.222.0.102 () {58 vars in 1014 bytes} [Wed Aug 24 15:46:48 2022] POST /identity/v3/auth/tokens => generated 3382 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:48.238622 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270ee936-1d14-4a4d-9f4e-a058873d9922 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:48.239218 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270ee936-1d14-4a4d-9f4e-a058873d9922 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:48.239789 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270ee936-1d14-4a4d-9f4e-a058873d9922 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:48.283822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-270ee936-1d14-4a4d-9f4e-a058873d9922 None None] MFA Rules not processed for user `9c5d4d66a1fa41ee9ae3b34f36ea594d`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:48.286175 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-270ee936-1d14-4a4d-9f4e-a058873d9922 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:48.401362 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 805/1639] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:48 2022] POST /identity/v3/auth/tokens => generated 3333 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:48.403139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270ee936-1d14-4a4d-9f4e-a058873d9922 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:48.410708 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d85bb0d-618d-48d0-9560-f1780cf01bea None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:48.422717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae94e6ea-dc7f-4f79-a74f-2c46407aff9a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:48.423326 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae94e6ea-dc7f-4f79-a74f-2c46407aff9a service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:48.423812 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae94e6ea-dc7f-4f79-a74f-2c46407aff9a service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:48.423994 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae94e6ea-dc7f-4f79-a74f-2c46407aff9a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:48.448998 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270ee936-1d14-4a4d-9f4e-a058873d9922 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:48.459849 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4514a22-b450-4006-ad46-851a9abac744 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:48.460721 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4514a22-b450-4006-ad46-851a9abac744 service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:48.461741 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4514a22-b450-4006-ad46-851a9abac744 service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:48.462161 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4514a22-b450-4006-ad46-851a9abac744 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:48.483192 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4514a22-b450-4006-ad46-851a9abac744 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:48.486732 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4514a22-b450-4006-ad46-851a9abac744 service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:48.506707 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 806/1640] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:46:48 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:46:48.576832 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae94e6ea-dc7f-4f79-a74f-2c46407aff9a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:48.578574 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae94e6ea-dc7f-4f79-a74f-2c46407aff9a service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:48.603505 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 835/1641] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:46:48 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 434 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:46:48.704751 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4514a22-b450-4006-ad46-851a9abac744 service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:48.735701 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4514a22-b450-4006-ad46-851a9abac744 service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:48.747577 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a098ec20-f07b-468c-bf6a-d1670d407125 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:48.748339 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a098ec20-f07b-468c-bf6a-d1670d407125 service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:48.748494 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a098ec20-f07b-468c-bf6a-d1670d407125 service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:48.748752 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a098ec20-f07b-468c-bf6a-d1670d407125 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:48.777902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a098ec20-f07b-468c-bf6a-d1670d407125 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:48.779407 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a098ec20-f07b-468c-bf6a-d1670d407125 service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:48.802515 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 807/1642] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:46:48 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:46:49.504568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae94e6ea-dc7f-4f79-a74f-2c46407aff9a service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:49.529537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae94e6ea-dc7f-4f79-a74f-2c46407aff9a service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:49.553378 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45d4b28-2694-477d-8c12-ac7d4839cc0a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:49.557734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45d4b28-2694-477d-8c12-ac7d4839cc0a service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:49.558053 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45d4b28-2694-477d-8c12-ac7d4839cc0a service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:49.558312 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45d4b28-2694-477d-8c12-ac7d4839cc0a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:49.580509 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45d4b28-2694-477d-8c12-ac7d4839cc0a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:49.583536 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45d4b28-2694-477d-8c12-ac7d4839cc0a service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:49.632572 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 836/1643] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:46:49 2022] GET /identity/v3/auth/tokens => generated 3333 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:46:50.605441 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a098ec20-f07b-468c-bf6a-d1670d407125 service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:50.622420 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a098ec20-f07b-468c-bf6a-d1670d407125 service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:50.647468 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a744a4-e125-4765-ad2b-b68aa5c9f0c8 tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c5d4d66a1fa41ee9ae3b34f36ea594d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd49879196fe94da7be642cdf7b0c4d35', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:50.648133 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a744a4-e125-4765-ad2b-b68aa5c9f0c8 tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:50.648424 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a744a4-e125-4765-ad2b-b68aa5c9f0c8 tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:50.649591 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a744a4-e125-4765-ad2b-b68aa5c9f0c8 tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:50.650245 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a744a4-e125-4765-ad2b-b68aa5c9f0c8 tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:46:50.650556 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a744a4-e125-4765-ad2b-b68aa5c9f0c8 tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:50.652323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a744a4-e125-4765-ad2b-b68aa5c9f0c8 tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:50.670795 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:46:50.683001 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 808/1644] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:46:50 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:46:50.686294 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45d4b28-2694-477d-8c12-ac7d4839cc0a service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:50.697190 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45d4b28-2694-477d-8c12-ac7d4839cc0a service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:50.706618 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a744a4-e125-4765-ad2b-b68aa5c9f0c8 tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:50.720116 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49acfd5b-25b1-482a-97ee-505c85dab1f1 tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c5d4d66a1fa41ee9ae3b34f36ea594d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd49879196fe94da7be642cdf7b0c4d35', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:50.720744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49acfd5b-25b1-482a-97ee-505c85dab1f1 tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:50.721029 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49acfd5b-25b1-482a-97ee-505c85dab1f1 tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:50.721890 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49acfd5b-25b1-482a-97ee-505c85dab1f1 tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:50.722433 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49acfd5b-25b1-482a-97ee-505c85dab1f1 tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:50.724476 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49acfd5b-25b1-482a-97ee-505c85dab1f1 tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:50.787725 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 837/1645] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:46:50 2022] POST /identity/v3/projects => generated 355 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:46:50.790267 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49acfd5b-25b1-482a-97ee-505c85dab1f1 tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:50.849426 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a744a4-e125-4765-ad2b-b68aa5c9f0c8 tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:50.932295 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49acfd5b-25b1-482a-97ee-505c85dab1f1 tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:50.939731 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c948ce1-6b1b-4a82-868e-ef0a8407515e tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c5d4d66a1fa41ee9ae3b34f36ea594d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd49879196fe94da7be642cdf7b0c4d35', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:50.941496 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c948ce1-6b1b-4a82-868e-ef0a8407515e tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:50.942046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c948ce1-6b1b-4a82-868e-ef0a8407515e tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:50.942472 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c948ce1-6b1b-4a82-868e-ef0a8407515e tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:50.943312 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c948ce1-6b1b-4a82-868e-ef0a8407515e tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:46:50.943996 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c948ce1-6b1b-4a82-868e-ef0a8407515e tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:50.946831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c948ce1-6b1b-4a82-868e-ef0a8407515e tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:50.964346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 838/1646] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:46:50 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:46:51.002830 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8ea0d6-e82b-4e0e-b166-e161b623eb34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:51.003084 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8ea0d6-e82b-4e0e-b166-e161b623eb34 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:51.003372 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8ea0d6-e82b-4e0e-b166-e161b623eb34 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:51.003698 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8ea0d6-e82b-4e0e-b166-e161b623eb34 admin admin] PATH_INFO: `/v3/users/ec9f1fb05ef1400abf74f1926d2ec385` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:51.004826 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8ea0d6-e82b-4e0e-b166-e161b623eb34 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ec9f1fb05ef1400abf74f1926d2ec385)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:51.008712 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: WARNING py.warnings [None req-ec8ea0d6-e82b-4e0e-b166-e161b623eb34 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 24 15:46:51.008712 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: warnings.warn(msg) Aug 24 15:46:51.008712 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: Aug 24 15:46:51.009969 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8ea0d6-e82b-4e0e-b166-e161b623eb34 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:51.025902 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c948ce1-6b1b-4a82-868e-ef0a8407515e tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:51.034445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:46:51.040688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c948ce1-6b1b-4a82-868e-ef0a8407515e tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:51.047954 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9d144c-b17d-4e29-9ecc-11f0f0a0f96a tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c5d4d66a1fa41ee9ae3b34f36ea594d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd49879196fe94da7be642cdf7b0c4d35', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:51.049483 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9d144c-b17d-4e29-9ecc-11f0f0a0f96a tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:51.050016 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9d144c-b17d-4e29-9ecc-11f0f0a0f96a tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:51.050502 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9d144c-b17d-4e29-9ecc-11f0f0a0f96a tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] PATH_INFO: `/v3/projects/c00f926ee9c24228806ed2e8bc674faf` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:51.051912 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9d144c-b17d-4e29-9ecc-11f0f0a0f96a tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] RBAC: Authorizing `identity:delete_project(project_id=c00f926ee9c24228806ed2e8bc674faf)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:51.053417 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: WARNING py.warnings [None req-dc9d144c-b17d-4e29-9ecc-11f0f0a0f96a tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 24 15:46:51.053417 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: warnings.warn(msg) Aug 24 15:46:51.053417 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: Aug 24 15:46:51.058397 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9d144c-b17d-4e29-9ecc-11f0f0a0f96a tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:51.159268 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-ec8ea0d6-e82b-4e0e-b166-e161b623eb34 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ec9f1fb05ef1400abf74f1926d2ec385'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:51.160086 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-ec8ea0d6-e82b-4e0e-b166-e161b623eb34 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ec9f1fb05ef1400abf74f1926d2ec385'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:51.187954 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-ec8ea0d6-e82b-4e0e-b166-e161b623eb34 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ec9f1fb05ef1400abf74f1926d2ec385'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:51.254252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-ec8ea0d6-e82b-4e0e-b166-e161b623eb34 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ec9f1fb05ef1400abf74f1926d2ec385'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:51.274793 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-ec8ea0d6-e82b-4e0e-b166-e161b623eb34 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ec9f1fb05ef1400abf74f1926d2ec385'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:51.340925 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 809/1647] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:46:50 2022] DELETE /identity/v3/users/ec9f1fb05ef1400abf74f1926d2ec385 => generated 0 bytes in 635 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 15:46:51.353356 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1faa6e-55cc-4a04-972d-e9b5a9f73701 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:51.353665 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1faa6e-55cc-4a04-972d-e9b5a9f73701 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:51.355436 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1faa6e-55cc-4a04-972d-e9b5a9f73701 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:51.366279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-dc9d144c-b17d-4e29-9ecc-11f0f0a0f96a tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c00f926ee9c24228806ed2e8bc674faf', 'request_id': 'req-dc9d144c-b17d-4e29-9ecc-11f0f0a0f96a'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:51.426630 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-dc9d144c-b17d-4e29-9ecc-11f0f0a0f96a tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c00f926ee9c24228806ed2e8bc674faf', 'request_id': 'req-dc9d144c-b17d-4e29-9ecc-11f0f0a0f96a'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:51.447328 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-ec1faa6e-55cc-4a04-972d-e9b5a9f73701 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:51.450412 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-ec1faa6e-55cc-4a04-972d-e9b5a9f73701 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:51.463405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-dc9d144c-b17d-4e29-9ecc-11f0f0a0f96a tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c00f926ee9c24228806ed2e8bc674faf', 'request_id': 'req-dc9d144c-b17d-4e29-9ecc-11f0f0a0f96a'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:51.495606 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-dc9d144c-b17d-4e29-9ecc-11f0f0a0f96a tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] The token cache is being invalidate because project c00f926ee9c24228806ed2e8bc674faf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:46:51.495981 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-dc9d144c-b17d-4e29-9ecc-11f0f0a0f96a tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:51.497029 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-dc9d144c-b17d-4e29-9ecc-11f0f0a0f96a tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:51.499228 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 839/1648] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:46:51 2022] DELETE /identity/v3/projects/c00f926ee9c24228806ed2e8bc674faf => generated 0 bytes in 474 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:46:51.509474 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:46:51.575563 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 810/1649] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:51 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:51.693866 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9d144c-b17d-4e29-9ecc-11f0f0a0f96a tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:51.699264 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-dc9d144c-b17d-4e29-9ecc-11f0f0a0f96a tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:51.889881 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9d144c-b17d-4e29-9ecc-11f0f0a0f96a tempest-VolumeQuotaClassesTest-437688528 tempest-VolumeQuotaClassesTest-437688528-project-admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:52.036487 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be44fcd7-5ea9-45ea-86d2-c19944b751ab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:52.037124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be44fcd7-5ea9-45ea-86d2-c19944b751ab service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:52.037556 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be44fcd7-5ea9-45ea-86d2-c19944b751ab service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:52.037886 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be44fcd7-5ea9-45ea-86d2-c19944b751ab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:52.051841 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be44fcd7-5ea9-45ea-86d2-c19944b751ab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:52.053323 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be44fcd7-5ea9-45ea-86d2-c19944b751ab service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:52.095508 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 840/1650] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:46:51 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:46:52.533327 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1faa6e-55cc-4a04-972d-e9b5a9f73701 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:52.534947 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-ec1faa6e-55cc-4a04-972d-e9b5a9f73701 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:52.549807 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1faa6e-55cc-4a04-972d-e9b5a9f73701 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:52.567941 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcea5cfc-e497-4c68-aff5-33e25a833dd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:52.569462 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcea5cfc-e497-4c68-aff5-33e25a833dd6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:52.569924 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcea5cfc-e497-4c68-aff5-33e25a833dd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:52.571497 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcea5cfc-e497-4c68-aff5-33e25a833dd6 admin admin] PATH_INFO: `/v3/users/29e13bb3a9584a0e8336224579fc2bfe` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:52.573969 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcea5cfc-e497-4c68-aff5-33e25a833dd6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=29e13bb3a9584a0e8336224579fc2bfe)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:52.577284 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcea5cfc-e497-4c68-aff5-33e25a833dd6 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:52.657992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-fcea5cfc-e497-4c68-aff5-33e25a833dd6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '29e13bb3a9584a0e8336224579fc2bfe'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:52.659196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-fcea5cfc-e497-4c68-aff5-33e25a833dd6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '29e13bb3a9584a0e8336224579fc2bfe'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:52.670180 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-fcea5cfc-e497-4c68-aff5-33e25a833dd6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '29e13bb3a9584a0e8336224579fc2bfe'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:52.698998 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-fcea5cfc-e497-4c68-aff5-33e25a833dd6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '29e13bb3a9584a0e8336224579fc2bfe'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:52.699805 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-fcea5cfc-e497-4c68-aff5-33e25a833dd6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '29e13bb3a9584a0e8336224579fc2bfe'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:52.725628 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 811/1651] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:46:52 2022] DELETE /identity/v3/users/29e13bb3a9584a0e8336224579fc2bfe => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:46:52.731392 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3900f97a-5527-4c81-bd29-7bf61c7915ea None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:52.732461 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3900f97a-5527-4c81-bd29-7bf61c7915ea None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:52.732752 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3900f97a-5527-4c81-bd29-7bf61c7915ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:52.769209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-3900f97a-5527-4c81-bd29-7bf61c7915ea None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:52.770950 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-3900f97a-5527-4c81-bd29-7bf61c7915ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:52.904446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:46:52.955093 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 841/1652] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:52 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:52.990839 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcea5cfc-e497-4c68-aff5-33e25a833dd6 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:52.993270 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-fcea5cfc-e497-4c68-aff5-33e25a833dd6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:53.118836 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcea5cfc-e497-4c68-aff5-33e25a833dd6 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:53.127229 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0bae07-9d1d-4f30-a0ed-547d7c1c7d93 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:53.127915 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0bae07-9d1d-4f30-a0ed-547d7c1c7d93 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:53.128196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0bae07-9d1d-4f30-a0ed-547d7c1c7d93 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:53.128860 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0bae07-9d1d-4f30-a0ed-547d7c1c7d93 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:53.148341 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b0bae07-9d1d-4f30-a0ed-547d7c1c7d93 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:53.150872 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b0bae07-9d1d-4f30-a0ed-547d7c1c7d93 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:53.175669 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 812/1653] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:46:52 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:46:54.197459 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3900f97a-5527-4c81-bd29-7bf61c7915ea None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:54.200038 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-3900f97a-5527-4c81-bd29-7bf61c7915ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:54.222800 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3900f97a-5527-4c81-bd29-7bf61c7915ea None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:54.245882 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b241cd-d4c4-455a-aba9-76b0569bdb51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:54.246902 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b241cd-d4c4-455a-aba9-76b0569bdb51 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:54.247389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b241cd-d4c4-455a-aba9-76b0569bdb51 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:54.248260 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b241cd-d4c4-455a-aba9-76b0569bdb51 admin admin] PATH_INFO: `/v3/projects/1e05694ccd044a8a88ce024ac2d0668d` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:54.249800 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b241cd-d4c4-455a-aba9-76b0569bdb51 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1e05694ccd044a8a88ce024ac2d0668d)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:54.255666 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b241cd-d4c4-455a-aba9-76b0569bdb51 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:54.368848 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-95b241cd-d4c4-455a-aba9-76b0569bdb51 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1e05694ccd044a8a88ce024ac2d0668d', 'request_id': 'req-95b241cd-d4c4-455a-aba9-76b0569bdb51'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:54.381461 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-95b241cd-d4c4-455a-aba9-76b0569bdb51 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1e05694ccd044a8a88ce024ac2d0668d', 'request_id': 'req-95b241cd-d4c4-455a-aba9-76b0569bdb51'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:54.403201 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-95b241cd-d4c4-455a-aba9-76b0569bdb51 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1e05694ccd044a8a88ce024ac2d0668d', 'request_id': 'req-95b241cd-d4c4-455a-aba9-76b0569bdb51'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:54.429068 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-95b241cd-d4c4-455a-aba9-76b0569bdb51 admin admin] The token cache is being invalidate because project 1e05694ccd044a8a88ce024ac2d0668d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:46:54.429368 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-95b241cd-d4c4-455a-aba9-76b0569bdb51 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:54.430651 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-95b241cd-d4c4-455a-aba9-76b0569bdb51 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:54.432303 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 842/1654] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:46:54 2022] DELETE /identity/v3/projects/1e05694ccd044a8a88ce024ac2d0668d => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:46:54.441822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0bae07-9d1d-4f30-a0ed-547d7c1c7d93 service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:54.457328 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0bae07-9d1d-4f30-a0ed-547d7c1c7d93 service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:54.571357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:46:54.618471 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83d86af-c4cd-4a3d-bb0e-d4cda795ea55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:54.619472 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83d86af-c4cd-4a3d-bb0e-d4cda795ea55 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:54.620984 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83d86af-c4cd-4a3d-bb0e-d4cda795ea55 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:54.621509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83d86af-c4cd-4a3d-bb0e-d4cda795ea55 admin admin] PATH_INFO: `/v3/users/71acccdbd5de4307b949471c4dd734a7` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:54.624159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c83d86af-c4cd-4a3d-bb0e-d4cda795ea55 admin admin] RBAC: Authorizing `identity:delete_user(user_id=71acccdbd5de4307b949471c4dd734a7)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:54.632266 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c83d86af-c4cd-4a3d-bb0e-d4cda795ea55 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:54.740217 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-c83d86af-c4cd-4a3d-bb0e-d4cda795ea55 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '71acccdbd5de4307b949471c4dd734a7'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:54.744148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-c83d86af-c4cd-4a3d-bb0e-d4cda795ea55 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '71acccdbd5de4307b949471c4dd734a7'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:54.761000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-c83d86af-c4cd-4a3d-bb0e-d4cda795ea55 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '71acccdbd5de4307b949471c4dd734a7'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:54.799560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-c83d86af-c4cd-4a3d-bb0e-d4cda795ea55 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '71acccdbd5de4307b949471c4dd734a7'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:54.800412 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-c83d86af-c4cd-4a3d-bb0e-d4cda795ea55 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '71acccdbd5de4307b949471c4dd734a7'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:54.835249 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 813/1655] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:46:54 2022] DELETE /identity/v3/users/71acccdbd5de4307b949471c4dd734a7 => generated 0 bytes in 394 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:46:55.607346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b241cd-d4c4-455a-aba9-76b0569bdb51 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:55.608974 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-95b241cd-d4c4-455a-aba9-76b0569bdb51 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:55.632443 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b241cd-d4c4-455a-aba9-76b0569bdb51 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:55.803164 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d5651a-a65e-4bd0-828e-b6adb9d716d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:55.804158 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d5651a-a65e-4bd0-828e-b6adb9d716d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:55.809001 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d5651a-a65e-4bd0-828e-b6adb9d716d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:55.809302 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d5651a-a65e-4bd0-828e-b6adb9d716d5 admin admin] PATH_INFO: `/v3/users/2f3ce0702d194ad9a78aed3a4fefe205` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:55.810715 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d5651a-a65e-4bd0-828e-b6adb9d716d5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2f3ce0702d194ad9a78aed3a4fefe205)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:55.812933 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d5651a-a65e-4bd0-828e-b6adb9d716d5 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:55.958791 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-37d5651a-a65e-4bd0-828e-b6adb9d716d5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2f3ce0702d194ad9a78aed3a4fefe205'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:55.959969 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-37d5651a-a65e-4bd0-828e-b6adb9d716d5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2f3ce0702d194ad9a78aed3a4fefe205'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:55.960705 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-37d5651a-a65e-4bd0-828e-b6adb9d716d5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2f3ce0702d194ad9a78aed3a4fefe205'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:55.983955 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-37d5651a-a65e-4bd0-828e-b6adb9d716d5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2f3ce0702d194ad9a78aed3a4fefe205'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:56.013531 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-37d5651a-a65e-4bd0-828e-b6adb9d716d5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2f3ce0702d194ad9a78aed3a4fefe205'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:56.017075 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:46:56.078708 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838a663c-4655-4e0f-a883-063d953a4a17 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:56.079326 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838a663c-4655-4e0f-a883-063d953a4a17 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:56.079326 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838a663c-4655-4e0f-a883-063d953a4a17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:56.079908 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 843/1656] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:46:55 2022] DELETE /identity/v3/users/2f3ce0702d194ad9a78aed3a4fefe205 => generated 0 bytes in 467 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:46:56.119597 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-838a663c-4655-4e0f-a883-063d953a4a17 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:56.121301 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-838a663c-4655-4e0f-a883-063d953a4a17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:56.264052 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 814/1657] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:56 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:56.277451 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d5651a-a65e-4bd0-828e-b6adb9d716d5 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:56.281302 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-37d5651a-a65e-4bd0-828e-b6adb9d716d5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:56.292265 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838a663c-4655-4e0f-a883-063d953a4a17 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:56.293959 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-838a663c-4655-4e0f-a883-063d953a4a17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:56.314790 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838a663c-4655-4e0f-a883-063d953a4a17 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:56.324322 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849e30fc-00a4-43f9-ae31-8f70673e4322 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:56.327069 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849e30fc-00a4-43f9-ae31-8f70673e4322 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:56.327956 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849e30fc-00a4-43f9-ae31-8f70673e4322 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:56.330759 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849e30fc-00a4-43f9-ae31-8f70673e4322 admin admin] PATH_INFO: `/v3/projects/82ee4cb082d14ccfba49cfa0ef25394e` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:56.335755 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-849e30fc-00a4-43f9-ae31-8f70673e4322 admin admin] RBAC: Authorizing `identity:delete_project(project_id=82ee4cb082d14ccfba49cfa0ef25394e)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:56.342112 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-849e30fc-00a4-43f9-ae31-8f70673e4322 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:56.450786 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-849e30fc-00a4-43f9-ae31-8f70673e4322 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '82ee4cb082d14ccfba49cfa0ef25394e', 'request_id': 'req-849e30fc-00a4-43f9-ae31-8f70673e4322'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:56.463354 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-849e30fc-00a4-43f9-ae31-8f70673e4322 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '82ee4cb082d14ccfba49cfa0ef25394e', 'request_id': 'req-849e30fc-00a4-43f9-ae31-8f70673e4322'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:56.468662 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:46:56.479987 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-849e30fc-00a4-43f9-ae31-8f70673e4322 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '82ee4cb082d14ccfba49cfa0ef25394e', 'request_id': 'req-849e30fc-00a4-43f9-ae31-8f70673e4322'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:56.485690 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d5651a-a65e-4bd0-828e-b6adb9d716d5 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:56.501831 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-849e30fc-00a4-43f9-ae31-8f70673e4322 admin admin] The token cache is being invalidate because project 82ee4cb082d14ccfba49cfa0ef25394e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:46:56.502213 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-849e30fc-00a4-43f9-ae31-8f70673e4322 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:56.503004 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-849e30fc-00a4-43f9-ae31-8f70673e4322 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:56.509994 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 815/1658] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:46:56 2022] DELETE /identity/v3/projects/82ee4cb082d14ccfba49cfa0ef25394e => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:46:56.511260 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849e30fc-00a4-43f9-ae31-8f70673e4322 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:56.515562 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-849e30fc-00a4-43f9-ae31-8f70673e4322 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:56.631268 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2235c161-2420-44f5-bc2c-8ad190faee19 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:56.631940 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2235c161-2420-44f5-bc2c-8ad190faee19 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:56.632308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2235c161-2420-44f5-bc2c-8ad190faee19 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:56.632623 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2235c161-2420-44f5-bc2c-8ad190faee19 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:56.662919 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-2235c161-2420-44f5-bc2c-8ad190faee19 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:56.729745 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849e30fc-00a4-43f9-ae31-8f70673e4322 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:56.744846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2053d8-91fa-4820-89ff-60bcea3b195f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:56.745503 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2053d8-91fa-4820-89ff-60bcea3b195f admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:56.746158 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2053d8-91fa-4820-89ff-60bcea3b195f admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:56.746823 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2053d8-91fa-4820-89ff-60bcea3b195f admin admin] PATH_INFO: `/v3/users/9c5d4d66a1fa41ee9ae3b34f36ea594d` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:56.752234 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d2053d8-91fa-4820-89ff-60bcea3b195f admin admin] RBAC: Authorizing `identity:delete_user(user_id=9c5d4d66a1fa41ee9ae3b34f36ea594d)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:56.763077 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d2053d8-91fa-4820-89ff-60bcea3b195f admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:56.864296 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2235c161-2420-44f5-bc2c-8ad190faee19 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:56.866795 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2235c161-2420-44f5-bc2c-8ad190faee19 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:56.886667 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 844/1659] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:46:56 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 610 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:46:56.889716 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7e0548-1912-44f1-8f9e-da508f7fb48c None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:56.890111 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7e0548-1912-44f1-8f9e-da508f7fb48c None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:56.890476 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7e0548-1912-44f1-8f9e-da508f7fb48c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:56.911460 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-5d2053d8-91fa-4820-89ff-60bcea3b195f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9c5d4d66a1fa41ee9ae3b34f36ea594d'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:56.912503 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-5d2053d8-91fa-4820-89ff-60bcea3b195f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9c5d4d66a1fa41ee9ae3b34f36ea594d'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:56.930378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-5d2053d8-91fa-4820-89ff-60bcea3b195f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9c5d4d66a1fa41ee9ae3b34f36ea594d'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:56.962612 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-5d2053d8-91fa-4820-89ff-60bcea3b195f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9c5d4d66a1fa41ee9ae3b34f36ea594d'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:56.962842 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-5d2053d8-91fa-4820-89ff-60bcea3b195f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9c5d4d66a1fa41ee9ae3b34f36ea594d'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:56.971478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-2e7e0548-1912-44f1-8f9e-da508f7fb48c None None] MFA Rules not processed for user `2fe5724620da46ddbefbdfd442dc8d73`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:56.971596 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-2e7e0548-1912-44f1-8f9e-da508f7fb48c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:57.008580 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 816/1660] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:46:56 2022] DELETE /identity/v3/users/9c5d4d66a1fa41ee9ae3b34f36ea594d => generated 0 bytes in 498 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:46:57.011255 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2053d8-91fa-4820-89ff-60bcea3b195f admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:57.042378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-5d2053d8-91fa-4820-89ff-60bcea3b195f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:57.092782 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:46:57.141893 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 845/1661] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:56 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 246 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:57.144856 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3331c9c2-9d67-41ba-9fd4-66bbe11242c8 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:57.144856 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3331c9c2-9d67-41ba-9fd4-66bbe11242c8 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:57.144856 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3331c9c2-9d67-41ba-9fd4-66bbe11242c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:57.187933 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-3331c9c2-9d67-41ba-9fd4-66bbe11242c8 None None] MFA Rules not processed for user `2fe5724620da46ddbefbdfd442dc8d73`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:57.191587 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-3331c9c2-9d67-41ba-9fd4-66bbe11242c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:57.302561 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 846/1662] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:57 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:57.302781 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2053d8-91fa-4820-89ff-60bcea3b195f admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:57.308910 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3331c9c2-9d67-41ba-9fd4-66bbe11242c8 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:57.310172 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-3331c9c2-9d67-41ba-9fd4-66bbe11242c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:57.314086 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46897b40-1bb4-40a2-a85e-f4d2e96da392 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:57.314837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46897b40-1bb4-40a2-a85e-f4d2e96da392 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:57.315876 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46897b40-1bb4-40a2-a85e-f4d2e96da392 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:57.317801 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46897b40-1bb4-40a2-a85e-f4d2e96da392 admin admin] PATH_INFO: `/v3/projects/708a7189013e4dfd80f0d51e04c8156c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:57.329613 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46897b40-1bb4-40a2-a85e-f4d2e96da392 admin admin] RBAC: Authorizing `identity:delete_project(project_id=708a7189013e4dfd80f0d51e04c8156c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:57.331905 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46897b40-1bb4-40a2-a85e-f4d2e96da392 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:57.354996 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:46:57.453415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-46897b40-1bb4-40a2-a85e-f4d2e96da392 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '708a7189013e4dfd80f0d51e04c8156c', 'request_id': 'req-46897b40-1bb4-40a2-a85e-f4d2e96da392'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:57.467964 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-46897b40-1bb4-40a2-a85e-f4d2e96da392 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '708a7189013e4dfd80f0d51e04c8156c', 'request_id': 'req-46897b40-1bb4-40a2-a85e-f4d2e96da392'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:57.472295 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3331c9c2-9d67-41ba-9fd4-66bbe11242c8 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:57.478213 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-46897b40-1bb4-40a2-a85e-f4d2e96da392 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '708a7189013e4dfd80f0d51e04c8156c', 'request_id': 'req-46897b40-1bb4-40a2-a85e-f4d2e96da392'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:57.516873 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-46897b40-1bb4-40a2-a85e-f4d2e96da392 admin admin] The token cache is being invalidate because project 708a7189013e4dfd80f0d51e04c8156c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:46:57.517166 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-46897b40-1bb4-40a2-a85e-f4d2e96da392 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:57.522600 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-46897b40-1bb4-40a2-a85e-f4d2e96da392 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:57.525536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 817/1663] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:46:57 2022] DELETE /identity/v3/projects/708a7189013e4dfd80f0d51e04c8156c => generated 0 bytes in 515 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:46:57.530637 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38253e6c-6e66-4088-999d-208d9711b938 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:57.530897 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38253e6c-6e66-4088-999d-208d9711b938 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:57.531160 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38253e6c-6e66-4088-999d-208d9711b938 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:57.585610 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-38253e6c-6e66-4088-999d-208d9711b938 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:57.592147 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-38253e6c-6e66-4088-999d-208d9711b938 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:57.646449 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daee647f-2b60-4cb5-ae4d-55ed57e94d72 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:57.647079 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daee647f-2b60-4cb5-ae4d-55ed57e94d72 service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:57.647357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daee647f-2b60-4cb5-ae4d-55ed57e94d72 service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:57.647642 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daee647f-2b60-4cb5-ae4d-55ed57e94d72 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:57.651741 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-daee647f-2b60-4cb5-ae4d-55ed57e94d72 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:57.709167 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:46:57.760208 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 818/1664] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:57 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:46:57.767911 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3518de37-e505-4d4d-a542-6faf0ab067c4 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:57.768308 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3518de37-e505-4d4d-a542-6faf0ab067c4 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:57.768587 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3518de37-e505-4d4d-a542-6faf0ab067c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:57.817413 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-3518de37-e505-4d4d-a542-6faf0ab067c4 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:57.818650 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-3518de37-e505-4d4d-a542-6faf0ab067c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:57.821180 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daee647f-2b60-4cb5-ae4d-55ed57e94d72 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:57.822882 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daee647f-2b60-4cb5-ae4d-55ed57e94d72 service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:57.838718 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 847/1665] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:46:57 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 530 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 15:46:57.843613 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 819/1666] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:57 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:46:57.848352 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daee647f-2b60-4cb5-ae4d-55ed57e94d72 service nova] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:57.872320 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daee647f-2b60-4cb5-ae4d-55ed57e94d72 service nova] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:57.888965 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e709a5f-35b5-4fb1-ad71-01abbd981ae2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:57.889602 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e709a5f-35b5-4fb1-ad71-01abbd981ae2 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:57.889881 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e709a5f-35b5-4fb1-ad71-01abbd981ae2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:57.890157 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e709a5f-35b5-4fb1-ad71-01abbd981ae2 admin admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:57.890859 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e709a5f-35b5-4fb1-ad71-01abbd981ae2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:46:57.891132 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e709a5f-35b5-4fb1-ad71-01abbd981ae2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:57.892852 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e709a5f-35b5-4fb1-ad71-01abbd981ae2 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:57.919537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 848/1667] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:46:57 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:46:57.923747 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3518de37-e505-4d4d-a542-6faf0ab067c4 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:57.935124 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3518de37-e505-4d4d-a542-6faf0ab067c4 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:57.971586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e78bcc1-add0-4015-afcc-b6686dc16ca8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:57.974860 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e78bcc1-add0-4015-afcc-b6686dc16ca8 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:57.975150 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e78bcc1-add0-4015-afcc-b6686dc16ca8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:57.975427 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e78bcc1-add0-4015-afcc-b6686dc16ca8 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:57.976143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e78bcc1-add0-4015-afcc-b6686dc16ca8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:57.978707 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e78bcc1-add0-4015-afcc-b6686dc16ca8 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:58.056499 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 820/1668] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:46:57 2022] POST /identity/v3/projects => generated 388 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:46:58.058733 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e709a5f-35b5-4fb1-ad71-01abbd981ae2 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:58.069491 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e709a5f-35b5-4fb1-ad71-01abbd981ae2 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:58.266490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8608000f-f40f-444c-a65b-3c82a37a4cde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:58.266943 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8608000f-f40f-444c-a65b-3c82a37a4cde admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:58.267227 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8608000f-f40f-444c-a65b-3c82a37a4cde admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:58.267498 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8608000f-f40f-444c-a65b-3c82a37a4cde admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:58.268170 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8608000f-f40f-444c-a65b-3c82a37a4cde admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:58.270388 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8608000f-f40f-444c-a65b-3c82a37a4cde admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:58.326763 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 849/1669] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:46:58 2022] POST /identity/v3/users => generated 295 bytes in 269 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:46:58.330874 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e78bcc1-add0-4015-afcc-b6686dc16ca8 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:58.349912 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e78bcc1-add0-4015-afcc-b6686dc16ca8 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:58.387906 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29120ef1-b1e4-4578-bdab-fb2669af2d85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:58.388547 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29120ef1-b1e4-4578-bdab-fb2669af2d85 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:58.388831 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29120ef1-b1e4-4578-bdab-fb2669af2d85 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:58.392829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29120ef1-b1e4-4578-bdab-fb2669af2d85 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:58.393543 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29120ef1-b1e4-4578-bdab-fb2669af2d85 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:58.405696 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29120ef1-b1e4-4578-bdab-fb2669af2d85 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:58.447487 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 821/1670] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:46:58 2022] GET /identity/v3/roles => generated 1427 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:46:58.454238 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8608000f-f40f-444c-a65b-3c82a37a4cde admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:58.476263 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8608000f-f40f-444c-a65b-3c82a37a4cde admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:58.496812 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83629f6e-8e46-455f-bd8a-ecf75bb5099b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:58.497442 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83629f6e-8e46-455f-bd8a-ecf75bb5099b admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:58.497732 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83629f6e-8e46-455f-bd8a-ecf75bb5099b admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:58.498164 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83629f6e-8e46-455f-bd8a-ecf75bb5099b admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:58.498844 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83629f6e-8e46-455f-bd8a-ecf75bb5099b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:58.504871 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83629f6e-8e46-455f-bd8a-ecf75bb5099b admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:58.554583 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29120ef1-b1e4-4578-bdab-fb2669af2d85 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:58.555324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 850/1671] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:46:58 2022] GET /identity/v3/roles => generated 1427 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:46:58.560065 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-29120ef1-b1e4-4578-bdab-fb2669af2d85 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:58.564999 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83629f6e-8e46-455f-bd8a-ecf75bb5099b admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:58.601026 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83629f6e-8e46-455f-bd8a-ecf75bb5099b admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:58.606337 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29120ef1-b1e4-4578-bdab-fb2669af2d85 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:58.622563 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da967f45-8132-4b21-b977-9e5a14f8f9f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:58.622781 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267f3679-d8fb-480e-87fe-47260c5966d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:58.623175 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da967f45-8132-4b21-b977-9e5a14f8f9f0 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:58.623522 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267f3679-d8fb-480e-87fe-47260c5966d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:58.624165 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267f3679-d8fb-480e-87fe-47260c5966d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:58.624560 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da967f45-8132-4b21-b977-9e5a14f8f9f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:58.624853 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da967f45-8132-4b21-b977-9e5a14f8f9f0 admin admin] PATH_INFO: `/v3/projects/d20a286c27cf4c8cbfbaffe60c2ad75f/users/afce420f59d2409598dc9e761eeaa1a2/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:58.624956 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267f3679-d8fb-480e-87fe-47260c5966d8 admin admin] PATH_INFO: `/v3/projects/d49879196fe94da7be642cdf7b0c4d35` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:58.630077 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267f3679-d8fb-480e-87fe-47260c5966d8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d49879196fe94da7be642cdf7b0c4d35)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:58.636256 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267f3679-d8fb-480e-87fe-47260c5966d8 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:58.677462 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da967f45-8132-4b21-b977-9e5a14f8f9f0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d20a286c27cf4c8cbfbaffe60c2ad75f, user_id=afce420f59d2409598dc9e761eeaa1a2, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:58.685630 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da967f45-8132-4b21-b977-9e5a14f8f9f0 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:58.728678 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 851/1672] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:46:58 2022] PUT /identity/v3/projects/d20a286c27cf4c8cbfbaffe60c2ad75f/users/afce420f59d2409598dc9e761eeaa1a2/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:46:58.738811 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da967f45-8132-4b21-b977-9e5a14f8f9f0 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:58.744074 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-da967f45-8132-4b21-b977-9e5a14f8f9f0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:58.783835 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-267f3679-d8fb-480e-87fe-47260c5966d8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd49879196fe94da7be642cdf7b0c4d35', 'request_id': 'req-267f3679-d8fb-480e-87fe-47260c5966d8'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:58.785100 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:46:58.855812 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-267f3679-d8fb-480e-87fe-47260c5966d8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd49879196fe94da7be642cdf7b0c4d35', 'request_id': 'req-267f3679-d8fb-480e-87fe-47260c5966d8'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:58.876834 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-267f3679-d8fb-480e-87fe-47260c5966d8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd49879196fe94da7be642cdf7b0c4d35', 'request_id': 'req-267f3679-d8fb-480e-87fe-47260c5966d8'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:58.925855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-267f3679-d8fb-480e-87fe-47260c5966d8 admin admin] The token cache is being invalidate because project d49879196fe94da7be642cdf7b0c4d35 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:46:58.926885 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-267f3679-d8fb-480e-87fe-47260c5966d8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:58.931079 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-267f3679-d8fb-480e-87fe-47260c5966d8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:58.942258 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 822/1673] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:46:58 2022] DELETE /identity/v3/projects/d49879196fe94da7be642cdf7b0c4d35 => generated 0 bytes in 384 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 15:46:58.944094 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6d1967-b56e-461b-b447-7cba7a185be4 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:58.944415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6d1967-b56e-461b-b447-7cba7a185be4 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:58.947717 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6d1967-b56e-461b-b447-7cba7a185be4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:58.989840 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da967f45-8132-4b21-b977-9e5a14f8f9f0 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:59.020603 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:46:59.115867 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-fd6d1967-b56e-461b-b447-7cba7a185be4 None None] MFA Rules not processed for user `afce420f59d2409598dc9e761eeaa1a2`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:59.123156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-fd6d1967-b56e-461b-b447-7cba7a185be4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:59.202785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b116affa-b1bf-4f36-98e7-093cffb7a5e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:59.203490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b116affa-b1bf-4f36-98e7-093cffb7a5e5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:59.205675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b116affa-b1bf-4f36-98e7-093cffb7a5e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:59.205836 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b116affa-b1bf-4f36-98e7-093cffb7a5e5 admin admin] PATH_INFO: `/v3/users/b7052b97e46e449685a65537f3b11864` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:59.207459 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b116affa-b1bf-4f36-98e7-093cffb7a5e5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b7052b97e46e449685a65537f3b11864)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:59.213320 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b116affa-b1bf-4f36-98e7-093cffb7a5e5 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:59.369600 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-b116affa-b1bf-4f36-98e7-093cffb7a5e5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b7052b97e46e449685a65537f3b11864'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:59.370493 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-b116affa-b1bf-4f36-98e7-093cffb7a5e5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b7052b97e46e449685a65537f3b11864'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:59.371230 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-b116affa-b1bf-4f36-98e7-093cffb7a5e5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b7052b97e46e449685a65537f3b11864'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:59.376686 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 823/1674] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:58 2022] POST /identity/v3/auth/tokens => generated 3274 bytes in 438 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:59.379646 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79740c8b-48df-4955-ae36-7f4d396eaf06 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:59.380447 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79740c8b-48df-4955-ae36-7f4d396eaf06 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:59.381279 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79740c8b-48df-4955-ae36-7f4d396eaf06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:59.393859 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-b116affa-b1bf-4f36-98e7-093cffb7a5e5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b7052b97e46e449685a65537f3b11864'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:59.435548 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-b116affa-b1bf-4f36-98e7-093cffb7a5e5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b7052b97e46e449685a65537f3b11864'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:46:59.455530 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-79740c8b-48df-4955-ae36-7f4d396eaf06 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:59.457700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-79740c8b-48df-4955-ae36-7f4d396eaf06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:59.467623 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 852/1675] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:46:58 2022] DELETE /identity/v3/users/b7052b97e46e449685a65537f3b11864 => generated 0 bytes in 738 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:46:59.468662 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b116affa-b1bf-4f36-98e7-093cffb7a5e5 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:59.482580 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 824/1676] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:59 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:59.487501 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a818ecf-ea2c-4d80-b42b-e25fd4ce4be6 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:59.488442 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a818ecf-ea2c-4d80-b42b-e25fd4ce4be6 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:59.489045 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a818ecf-ea2c-4d80-b42b-e25fd4ce4be6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:59.496532 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-b116affa-b1bf-4f36-98e7-093cffb7a5e5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:59.542047 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-9a818ecf-ea2c-4d80-b42b-e25fd4ce4be6 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:59.545390 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-9a818ecf-ea2c-4d80-b42b-e25fd4ce4be6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:59.674159 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b116affa-b1bf-4f36-98e7-093cffb7a5e5 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:59.679429 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3cd6186-39ff-4aac-b848-392fd66364e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:59.680188 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3cd6186-39ff-4aac-b848-392fd66364e5 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:59.680724 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3cd6186-39ff-4aac-b848-392fd66364e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:59.680901 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3cd6186-39ff-4aac-b848-392fd66364e5 admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:59.681551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3cd6186-39ff-4aac-b848-392fd66364e5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:59.683661 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3cd6186-39ff-4aac-b848-392fd66364e5 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:59.725772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 853/1677] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:46:59 2022] POST /identity/v3/projects => generated 386 bytes in 255 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:46:59.725772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70552efc-26d3-4a0a-8729-39de9649350a None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:59.725772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70552efc-26d3-4a0a-8729-39de9649350a None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:59.725772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70552efc-26d3-4a0a-8729-39de9649350a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:59.764535 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 825/1678] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:59 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 281 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:46:59.766929 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a818ecf-ea2c-4d80-b42b-e25fd4ce4be6 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:46:59.790361 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a818ecf-ea2c-4d80-b42b-e25fd4ce4be6 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:46:59.798144 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-70552efc-26d3-4a0a-8729-39de9649350a None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:46:59.799573 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-70552efc-26d3-4a0a-8729-39de9649350a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:46:59.839537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:46:59.959226 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1c8073-b40a-4a53-9bd0-46c45821d6c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:46:59.960868 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1c8073-b40a-4a53-9bd0-46c45821d6c4 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:46:59.962019 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1c8073-b40a-4a53-9bd0-46c45821d6c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:46:59.964130 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1c8073-b40a-4a53-9bd0-46c45821d6c4 admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:46:59.965010 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef1c8073-b40a-4a53-9bd0-46c45821d6c4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:46:59.967136 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef1c8073-b40a-4a53-9bd0-46c45821d6c4 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:46:59.993028 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=95,keystone:DELETE=3,keystone:INSERT=2 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:00.007342 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 854/1679] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:46:59 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 283 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:00.009356 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70552efc-26d3-4a0a-8729-39de9649350a None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:00.013547 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-70552efc-26d3-4a0a-8729-39de9649350a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:00.016704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 826/1680] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:46:59 2022] POST /identity/v3/users => generated 300 bytes in 251 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:00.017695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1c8073-b40a-4a53-9bd0-46c45821d6c4 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:00.045701 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1c8073-b40a-4a53-9bd0-46c45821d6c4 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:00.071118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e81ed0-2f17-4d64-9f00-3305fc29e1d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:00.074173 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e81ed0-2f17-4d64-9f00-3305fc29e1d1 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:00.074473 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e81ed0-2f17-4d64-9f00-3305fc29e1d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:00.074864 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e81ed0-2f17-4d64-9f00-3305fc29e1d1 admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:00.075684 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e81ed0-2f17-4d64-9f00-3305fc29e1d1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:47:00.076949 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e81ed0-2f17-4d64-9f00-3305fc29e1d1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:00.079925 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e81ed0-2f17-4d64-9f00-3305fc29e1d1 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:00.114354 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 827/1681] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:47:00 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:47:00.114354 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e81ed0-2f17-4d64-9f00-3305fc29e1d1 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:00.128763 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e81ed0-2f17-4d64-9f00-3305fc29e1d1 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:00.140751 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c716021-fb65-4584-bd82-70a7b7c95259 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:00.141883 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c716021-fb65-4584-bd82-70a7b7c95259 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:00.142304 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c716021-fb65-4584-bd82-70a7b7c95259 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:00.142819 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c716021-fb65-4584-bd82-70a7b7c95259 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:00.143662 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c716021-fb65-4584-bd82-70a7b7c95259 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:00.145734 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c716021-fb65-4584-bd82-70a7b7c95259 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:00.168784 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 828/1682] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:00 2022] GET /identity/v3/roles => generated 1427 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:00.170301 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c716021-fb65-4584-bd82-70a7b7c95259 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:00.183008 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c716021-fb65-4584-bd82-70a7b7c95259 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:00.190444 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a6a9c8-f731-463d-ae3d-caec7f519360 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:00.191542 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a6a9c8-f731-463d-ae3d-caec7f519360 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:00.191972 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a6a9c8-f731-463d-ae3d-caec7f519360 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:00.192511 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a6a9c8-f731-463d-ae3d-caec7f519360 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:00.193319 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a6a9c8-f731-463d-ae3d-caec7f519360 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:00.195987 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a6a9c8-f731-463d-ae3d-caec7f519360 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:00.244484 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 829/1683] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:47:00 2022] POST /identity/v3/projects => generated 380 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:00.245684 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a6a9c8-f731-463d-ae3d-caec7f519360 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:00.252227 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70552efc-26d3-4a0a-8729-39de9649350a None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:00.365050 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a6a9c8-f731-463d-ae3d-caec7f519360 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:00.380850 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49e6161-2753-4038-bc6a-3eab6f532b78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:00.382950 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49e6161-2753-4038-bc6a-3eab6f532b78 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:00.383470 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49e6161-2753-4038-bc6a-3eab6f532b78 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:00.384252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49e6161-2753-4038-bc6a-3eab6f532b78 admin admin] PATH_INFO: `/v3/projects/e1dc926c14aa40d18e0cf91c1e417bde/users/de56a2b58a8146aebb18ade78a98ea82/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:00.390039 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fdb5f96-bc42-453f-94af-a837944ad46c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:00.390039 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdb5f96-bc42-453f-94af-a837944ad46c service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:00.390458 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdb5f96-bc42-453f-94af-a837944ad46c service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:00.390590 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdb5f96-bc42-453f-94af-a837944ad46c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:00.407676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49e6161-2753-4038-bc6a-3eab6f532b78 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e1dc926c14aa40d18e0cf91c1e417bde, user_id=de56a2b58a8146aebb18ade78a98ea82, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:00.409959 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fdb5f96-bc42-453f-94af-a837944ad46c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:00.411489 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fdb5f96-bc42-453f-94af-a837944ad46c service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:00.412837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49e6161-2753-4038-bc6a-3eab6f532b78 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:00.440875 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 855/1684] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:00 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:00.443663 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fdb5f96-bc42-453f-94af-a837944ad46c service neutron] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:00.459221 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 830/1685] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:47:00 2022] PUT /identity/v3/projects/e1dc926c14aa40d18e0cf91c1e417bde/users/de56a2b58a8146aebb18ade78a98ea82/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:00.462325 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49e6161-2753-4038-bc6a-3eab6f532b78 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:00.464121 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fdb5f96-bc42-453f-94af-a837944ad46c service neutron] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:00.583009 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84153865-6e92-41da-9533-09c2f1558ffe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:00.583766 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84153865-6e92-41da-9533-09c2f1558ffe admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:00.584044 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84153865-6e92-41da-9533-09c2f1558ffe admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:00.584313 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84153865-6e92-41da-9533-09c2f1558ffe admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:00.585491 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84153865-6e92-41da-9533-09c2f1558ffe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:00.588067 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84153865-6e92-41da-9533-09c2f1558ffe admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:00.637118 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 856/1686] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:47:00 2022] POST /identity/v3/users => generated 291 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:00.638417 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84153865-6e92-41da-9533-09c2f1558ffe admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:00.657116 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84153865-6e92-41da-9533-09c2f1558ffe admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:00.666655 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49e6161-2753-4038-bc6a-3eab6f532b78 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:00.667463 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc11418c-30ae-49ba-8423-8bb8ee9d7c0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:00.668975 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc11418c-30ae-49ba-8423-8bb8ee9d7c0a admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:00.669266 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc11418c-30ae-49ba-8423-8bb8ee9d7c0a admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:00.669537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc11418c-30ae-49ba-8423-8bb8ee9d7c0a admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:00.671441 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc11418c-30ae-49ba-8423-8bb8ee9d7c0a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:00.679054 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc11418c-30ae-49ba-8423-8bb8ee9d7c0a admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:00.680679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f603b1dc-1012-4709-a899-7b9ad1c5354c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:00.683616 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:00.684243 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f603b1dc-1012-4709-a899-7b9ad1c5354c admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:00.688416 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f603b1dc-1012-4709-a899-7b9ad1c5354c admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:00.688954 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f603b1dc-1012-4709-a899-7b9ad1c5354c admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:00.693507 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f603b1dc-1012-4709-a899-7b9ad1c5354c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:00.696001 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f603b1dc-1012-4709-a899-7b9ad1c5354c admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:00.713430 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 857/1687] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:00 2022] GET /identity/v3/roles => generated 1427 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:00.722193 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc11418c-30ae-49ba-8423-8bb8ee9d7c0a admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:00.733595 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 831/1688] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:00 2022] GET /identity/v3/roles => generated 1427 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:00.739323 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc11418c-30ae-49ba-8423-8bb8ee9d7c0a admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:00.740624 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f603b1dc-1012-4709-a899-7b9ad1c5354c admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:00.750931 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c280e562-b983-48dc-8ad7-b62f8daebbd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:00.752188 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c280e562-b983-48dc-8ad7-b62f8daebbd7 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:00.752550 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c280e562-b983-48dc-8ad7-b62f8daebbd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:00.752892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c280e562-b983-48dc-8ad7-b62f8daebbd7 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:00.756598 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c280e562-b983-48dc-8ad7-b62f8daebbd7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:00.759517 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c280e562-b983-48dc-8ad7-b62f8daebbd7 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:00.772065 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f603b1dc-1012-4709-a899-7b9ad1c5354c admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:00.788619 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 858/1689] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:00 2022] GET /identity/v3/roles => generated 1427 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 15:47:00.791040 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c280e562-b983-48dc-8ad7-b62f8daebbd7 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:00.801110 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edec7f12-4c38-4c68-9608-518b855a4b91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:00.802043 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edec7f12-4c38-4c68-9608-518b855a4b91 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:00.802463 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edec7f12-4c38-4c68-9608-518b855a4b91 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:00.802746 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edec7f12-4c38-4c68-9608-518b855a4b91 admin admin] PATH_INFO: `/v3/projects/e1dc926c14aa40d18e0cf91c1e417bde/users/de56a2b58a8146aebb18ade78a98ea82/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:00.813429 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edec7f12-4c38-4c68-9608-518b855a4b91 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e1dc926c14aa40d18e0cf91c1e417bde, user_id=de56a2b58a8146aebb18ade78a98ea82, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:00.815407 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c280e562-b983-48dc-8ad7-b62f8daebbd7 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:00.816967 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edec7f12-4c38-4c68-9608-518b855a4b91 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:00.826283 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b076b1-e1c7-455f-8313-007add7e055b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:00.833447 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b076b1-e1c7-455f-8313-007add7e055b admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:00.833967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b076b1-e1c7-455f-8313-007add7e055b admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:00.834390 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b076b1-e1c7-455f-8313-007add7e055b admin admin] PATH_INFO: `/v3/projects/bf1040f6c7a3412dbf0cb067e736c2cc/users/eb6e8939a43c4b52b81d13351a108adb/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:00.844929 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 832/1690] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:47:00 2022] PUT /identity/v3/projects/e1dc926c14aa40d18e0cf91c1e417bde/users/de56a2b58a8146aebb18ade78a98ea82/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:00.855410 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a7443c-7603-4f3a-9232-b89039ca5bec None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:00.855749 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a7443c-7603-4f3a-9232-b89039ca5bec None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:00.855997 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a7443c-7603-4f3a-9232-b89039ca5bec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:00.869569 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b076b1-e1c7-455f-8313-007add7e055b admin admin] RBAC: Authorizing `identity:create_grant(project_id=bf1040f6c7a3412dbf0cb067e736c2cc, user_id=eb6e8939a43c4b52b81d13351a108adb, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:00.872361 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b076b1-e1c7-455f-8313-007add7e055b admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:00.896534 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:00.916053 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-c2a7443c-7603-4f3a-9232-b89039ca5bec None None] MFA Rules not processed for user `de56a2b58a8146aebb18ade78a98ea82`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:00.917913 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-c2a7443c-7603-4f3a-9232-b89039ca5bec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:00.922560 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 859/1691] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:47:00 2022] PUT /identity/v3/projects/bf1040f6c7a3412dbf0cb067e736c2cc/users/eb6e8939a43c4b52b81d13351a108adb/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:00.925050 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8793b013-130a-4495-a6e4-9d30a1cfd247 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:00.925347 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8793b013-130a-4495-a6e4-9d30a1cfd247 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:00.925627 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8793b013-130a-4495-a6e4-9d30a1cfd247 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:00.996396 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-8793b013-130a-4495-a6e4-9d30a1cfd247 None None] MFA Rules not processed for user `9db03e18a7b24157a8ac724a615373f4`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:00.997654 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-8793b013-130a-4495-a6e4-9d30a1cfd247 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:01.137968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 833/1692] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:00 2022] POST /identity/v3/auth/tokens => generated 3339 bytes in 283 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:01.139753 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090328d6-2528-4167-ba36-32828bdcdee6 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:01.140667 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090328d6-2528-4167-ba36-32828bdcdee6 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:01.141081 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090328d6-2528-4167-ba36-32828bdcdee6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:01.159579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 860/1693] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:00 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 236 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:01.176312 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f704801-668a-4b07-b504-3b44700d9abc None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:01.176662 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f704801-668a-4b07-b504-3b44700d9abc None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:01.177001 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f704801-668a-4b07-b504-3b44700d9abc None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:01.182303 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 861/1694] 10.222.0.102 () {54 vars in 917 bytes} [Wed Aug 24 15:47:01 2022] GET /identity => generated 271 bytes in 22 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:47:01.183376 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f704801-668a-4b07-b504-3b44700d9abc None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:01.192435 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-0f704801-668a-4b07-b504-3b44700d9abc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:01.224228 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-090328d6-2528-4167-ba36-32828bdcdee6 None None] MFA Rules not processed for user `eb6e8939a43c4b52b81d13351a108adb`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:01.225484 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-090328d6-2528-4167-ba36-32828bdcdee6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:01.352201 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f704801-668a-4b07-b504-3b44700d9abc None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:01.364574 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 834/1695] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:01 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:01.365462 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46b880d-edad-42a9-b4d1-8e72173f07e0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:01.367035 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b59f0f1-4fbd-47ea-a4e7-538ff03b70cb None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:01.368444 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46b880d-edad-42a9-b4d1-8e72173f07e0 service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:01.368886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b59f0f1-4fbd-47ea-a4e7-538ff03b70cb None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:01.369325 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46b880d-edad-42a9-b4d1-8e72173f07e0 service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:01.369733 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b59f0f1-4fbd-47ea-a4e7-538ff03b70cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:01.370138 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46b880d-edad-42a9-b4d1-8e72173f07e0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:01.397457 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b46b880d-edad-42a9-b4d1-8e72173f07e0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:01.400313 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b46b880d-edad-42a9-b4d1-8e72173f07e0 service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:01.417357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-3b59f0f1-4fbd-47ea-a4e7-538ff03b70cb None None] MFA Rules not processed for user `9db03e18a7b24157a8ac724a615373f4`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:01.418725 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-3b59f0f1-4fbd-47ea-a4e7-538ff03b70cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:01.434445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 862/1696] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:01 2022] GET /identity/v3/auth/tokens => generated 3339 bytes in 251 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:01.435610 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5463a470-92c7-4d76-a105-9dcc2cfd4321 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:01.436043 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5463a470-92c7-4d76-a105-9dcc2cfd4321 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:01.436361 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5463a470-92c7-4d76-a105-9dcc2cfd4321 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:01.447519 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 835/1697] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:01 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:01.448947 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b59f0f1-4fbd-47ea-a4e7-538ff03b70cb None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:01.458599 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-3b59f0f1-4fbd-47ea-a4e7-538ff03b70cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:01.560221 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-5463a470-92c7-4d76-a105-9dcc2cfd4321 None None] MFA Rules not processed for user `48939674c1fd4cf599691d510fcd8858`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:01.561867 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-5463a470-92c7-4d76-a105-9dcc2cfd4321 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:01.701778 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b59f0f1-4fbd-47ea-a4e7-538ff03b70cb None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:01.707744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:01.728447 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f346300-c1fa-4f77-8a00-39fba5284531 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:01.729463 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f346300-c1fa-4f77-8a00-39fba5284531 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:01.730045 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f346300-c1fa-4f77-8a00-39fba5284531 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:01.731152 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f346300-c1fa-4f77-8a00-39fba5284531 admin admin] PATH_INFO: `/v3/projects/e64b544bd42945e8bc11abeb9cf4cf0a` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:01.733502 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f346300-c1fa-4f77-8a00-39fba5284531 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e64b544bd42945e8bc11abeb9cf4cf0a)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:01.738756 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f346300-c1fa-4f77-8a00-39fba5284531 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:01.776233 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 863/1698] 10.222.0.102 () {58 vars in 1014 bytes} [Wed Aug 24 15:47:01 2022] POST /identity/v3/auth/tokens => generated 3131 bytes in 341 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:01.784662 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5463a470-92c7-4d76-a105-9dcc2cfd4321 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:01.816046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5463a470-92c7-4d76-a105-9dcc2cfd4321 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:01.829945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76406fbc-b3ca-4164-ba64-1b9279e04f91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:01.830582 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76406fbc-b3ca-4164-ba64-1b9279e04f91 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:01.830863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76406fbc-b3ca-4164-ba64-1b9279e04f91 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:01.831138 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76406fbc-b3ca-4164-ba64-1b9279e04f91 admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:01.840046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76406fbc-b3ca-4164-ba64-1b9279e04f91 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:01.845487 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76406fbc-b3ca-4164-ba64-1b9279e04f91 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:01.881530 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:01.946139 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 864/1699] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:47:01 2022] POST /identity/v3/projects => generated 378 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:01.947390 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76406fbc-b3ca-4164-ba64-1b9279e04f91 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:01.949961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-76406fbc-b3ca-4164-ba64-1b9279e04f91 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:01.965459 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-9f346300-c1fa-4f77-8a00-39fba5284531 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e64b544bd42945e8bc11abeb9cf4cf0a', 'request_id': 'req-9f346300-c1fa-4f77-8a00-39fba5284531'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:01.975288 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-9f346300-c1fa-4f77-8a00-39fba5284531 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e64b544bd42945e8bc11abeb9cf4cf0a', 'request_id': 'req-9f346300-c1fa-4f77-8a00-39fba5284531'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:01.997378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-9f346300-c1fa-4f77-8a00-39fba5284531 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e64b544bd42945e8bc11abeb9cf4cf0a', 'request_id': 'req-9f346300-c1fa-4f77-8a00-39fba5284531'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:02.025605 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-9f346300-c1fa-4f77-8a00-39fba5284531 admin admin] The token cache is being invalidate because project e64b544bd42945e8bc11abeb9cf4cf0a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:47:02.025922 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-9f346300-c1fa-4f77-8a00-39fba5284531 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:02.030676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-9f346300-c1fa-4f77-8a00-39fba5284531 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:02.036076 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 836/1700] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:47:01 2022] DELETE /identity/v3/projects/e64b544bd42945e8bc11abeb9cf4cf0a => generated 0 bytes in 585 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:02.036076 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f346300-c1fa-4f77-8a00-39fba5284531 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:02.037763 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-9f346300-c1fa-4f77-8a00-39fba5284531 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:02.130317 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76406fbc-b3ca-4164-ba64-1b9279e04f91 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:02.147280 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f61de736-770e-401d-ba1f-8182c68dc18b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:02.148381 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61de736-770e-401d-ba1f-8182c68dc18b service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:02.149233 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61de736-770e-401d-ba1f-8182c68dc18b service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:02.149506 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61de736-770e-401d-ba1f-8182c68dc18b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:02.155243 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-f61de736-770e-401d-ba1f-8182c68dc18b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:02.189272 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f346300-c1fa-4f77-8a00-39fba5284531 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:02.298006 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f61de736-770e-401d-ba1f-8182c68dc18b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:02.299740 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f61de736-770e-401d-ba1f-8182c68dc18b service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:02.302579 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973cc889-5581-4a6f-a21d-85a3c57b0635 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:02.303467 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973cc889-5581-4a6f-a21d-85a3c57b0635 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:02.304317 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973cc889-5581-4a6f-a21d-85a3c57b0635 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:02.306044 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973cc889-5581-4a6f-a21d-85a3c57b0635 admin admin] PATH_INFO: `/v3/users/afce420f59d2409598dc9e761eeaa1a2` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:02.308216 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-973cc889-5581-4a6f-a21d-85a3c57b0635 admin admin] RBAC: Authorizing `identity:delete_user(user_id=afce420f59d2409598dc9e761eeaa1a2)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:02.310894 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-973cc889-5581-4a6f-a21d-85a3c57b0635 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:02.311832 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 865/1701] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:01 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:02.313070 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f61de736-770e-401d-ba1f-8182c68dc18b service nova] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:02.315088 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-f61de736-770e-401d-ba1f-8182c68dc18b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:02.450103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-973cc889-5581-4a6f-a21d-85a3c57b0635 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'afce420f59d2409598dc9e761eeaa1a2'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:02.451004 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-973cc889-5581-4a6f-a21d-85a3c57b0635 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'afce420f59d2409598dc9e761eeaa1a2'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:02.475599 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-973cc889-5581-4a6f-a21d-85a3c57b0635 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'afce420f59d2409598dc9e761eeaa1a2'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:02.484658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:DELETE=14,keystone:SELECT=85,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:02.505333 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f61de736-770e-401d-ba1f-8182c68dc18b service nova] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:02.508847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-973cc889-5581-4a6f-a21d-85a3c57b0635 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'afce420f59d2409598dc9e761eeaa1a2'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:02.512107 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-973cc889-5581-4a6f-a21d-85a3c57b0635 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'afce420f59d2409598dc9e761eeaa1a2'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:02.518596 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a68bbe-7a94-4109-b3b3-e66e32fe4d22 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:02.519723 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a68bbe-7a94-4109-b3b3-e66e32fe4d22 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:02.520152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a68bbe-7a94-4109-b3b3-e66e32fe4d22 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:02.520863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a68bbe-7a94-4109-b3b3-e66e32fe4d22 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:02.528207 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-69a68bbe-7a94-4109-b3b3-e66e32fe4d22 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:02.541143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 837/1702] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:47:02 2022] DELETE /identity/v3/users/afce420f59d2409598dc9e761eeaa1a2 => generated 0 bytes in 507 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:02.542422 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973cc889-5581-4a6f-a21d-85a3c57b0635 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:02.544658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-973cc889-5581-4a6f-a21d-85a3c57b0635 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:02.546294 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a68bbe-7a94-4109-b3b3-e66e32fe4d22 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:02.548405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a68bbe-7a94-4109-b3b3-e66e32fe4d22 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:02.566092 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 866/1703] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:02 2022] GET /identity/v3/auth/tokens => generated 3131 bytes in 253 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:02.568273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047a3ca6-4dca-4efa-8dee-116f2d02468a None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:02.570167 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047a3ca6-4dca-4efa-8dee-116f2d02468a None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:02.571223 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047a3ca6-4dca-4efa-8dee-116f2d02468a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:02.611361 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:02.638922 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-047a3ca6-4dca-4efa-8dee-116f2d02468a None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:02.641125 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-047a3ca6-4dca-4efa-8dee-116f2d02468a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:02.713283 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973cc889-5581-4a6f-a21d-85a3c57b0635 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:02.726256 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b49d75-7915-4223-ad4d-0383cf2e538c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:02.727151 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b49d75-7915-4223-ad4d-0383cf2e538c admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:02.727433 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b49d75-7915-4223-ad4d-0383cf2e538c admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:02.728172 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b49d75-7915-4223-ad4d-0383cf2e538c admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:02.728831 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b49d75-7915-4223-ad4d-0383cf2e538c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:02.734126 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b49d75-7915-4223-ad4d-0383cf2e538c admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:02.757394 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 867/1704] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:02 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:02.758515 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0faa59-ca7a-4d03-959d-c3118d0f72d6 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:02.758515 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0faa59-ca7a-4d03-959d-c3118d0f72d6 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:02.758984 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0faa59-ca7a-4d03-959d-c3118d0f72d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:02.780098 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 838/1705] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:47:02 2022] POST /identity/v3/users => generated 290 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:02.784179 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab53578-edbc-4e72-bb6e-23254350bf4f None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:02.787080 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab53578-edbc-4e72-bb6e-23254350bf4f None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:02.787371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab53578-edbc-4e72-bb6e-23254350bf4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:02.814858 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-3f0faa59-ca7a-4d03-959d-c3118d0f72d6 None None] MFA Rules not processed for user `c058b07c1e9e443ebacbf38b1f13e58b`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:02.817789 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-3f0faa59-ca7a-4d03-959d-c3118d0f72d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:02.851363 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-8ab53578-edbc-4e72-bb6e-23254350bf4f None None] MFA Rules not processed for user `38f7d31cf0f541f58af7d3d850af6472`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:02.853159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-8ab53578-edbc-4e72-bb6e-23254350bf4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:03.005230 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 868/1706] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:02 2022] POST /identity/v3/auth/tokens => generated 3274 bytes in 252 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:03.007206 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0faa59-ca7a-4d03-959d-c3118d0f72d6 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:03.015659 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-3f0faa59-ca7a-4d03-959d-c3118d0f72d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:03.051660 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0faa59-ca7a-4d03-959d-c3118d0f72d6 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:03.062200 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 839/1707] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:02 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 273 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:03.062200 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff65bdc-5475-4369-956c-89e31c7d3a55 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:03.062200 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff65bdc-5475-4369-956c-89e31c7d3a55 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:03.062200 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff65bdc-5475-4369-956c-89e31c7d3a55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:03.066175 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a369ab7f-6b4e-4925-b478-8e8b66956d1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:03.067089 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a369ab7f-6b4e-4925-b478-8e8b66956d1e admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:03.067367 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a369ab7f-6b4e-4925-b478-8e8b66956d1e admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:03.070317 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a369ab7f-6b4e-4925-b478-8e8b66956d1e admin admin] PATH_INFO: `/v3/projects/42da093227fd487c8df40a14d4a38c1a` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:03.072927 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a369ab7f-6b4e-4925-b478-8e8b66956d1e admin admin] RBAC: Authorizing `identity:delete_project(project_id=42da093227fd487c8df40a14d4a38c1a)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:03.079019 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a369ab7f-6b4e-4925-b478-8e8b66956d1e admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:03.091504 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-4ff65bdc-5475-4369-956c-89e31c7d3a55 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:03.093457 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-4ff65bdc-5475-4369-956c-89e31c7d3a55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:03.103245 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 840/1708] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:03 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:03.104919 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89eb96ab-f417-4eec-b2cb-2c335d5166a4 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:03.105411 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89eb96ab-f417-4eec-b2cb-2c335d5166a4 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:03.106015 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89eb96ab-f417-4eec-b2cb-2c335d5166a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:03.151381 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-89eb96ab-f417-4eec-b2cb-2c335d5166a4 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:03.153760 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-89eb96ab-f417-4eec-b2cb-2c335d5166a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:03.163444 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 841/1709] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:03 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:03.168301 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89eb96ab-f417-4eec-b2cb-2c335d5166a4 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:03.186320 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89eb96ab-f417-4eec-b2cb-2c335d5166a4 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:03.197008 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-a369ab7f-6b4e-4925-b478-8e8b66956d1e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '42da093227fd487c8df40a14d4a38c1a', 'request_id': 'req-a369ab7f-6b4e-4925-b478-8e8b66956d1e'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:03.214499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-a369ab7f-6b4e-4925-b478-8e8b66956d1e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '42da093227fd487c8df40a14d4a38c1a', 'request_id': 'req-a369ab7f-6b4e-4925-b478-8e8b66956d1e'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:03.226554 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-a369ab7f-6b4e-4925-b478-8e8b66956d1e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '42da093227fd487c8df40a14d4a38c1a', 'request_id': 'req-a369ab7f-6b4e-4925-b478-8e8b66956d1e'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:03.252038 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-a369ab7f-6b4e-4925-b478-8e8b66956d1e admin admin] The token cache is being invalidate because project 42da093227fd487c8df40a14d4a38c1a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:47:03.252307 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-a369ab7f-6b4e-4925-b478-8e8b66956d1e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:03.255105 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-a369ab7f-6b4e-4925-b478-8e8b66956d1e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:03.259089 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 869/1710] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:47:03 2022] DELETE /identity/v3/projects/42da093227fd487c8df40a14d4a38c1a => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:03.263510 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a369ab7f-6b4e-4925-b478-8e8b66956d1e admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:03.266752 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-a369ab7f-6b4e-4925-b478-8e8b66956d1e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:03.346697 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:03.360531 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=93,keystone:DELETE=3,keystone:INSERT=4 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:03.398482 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acff95e4-596b-420f-8576-a47ded494913 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:03.399997 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acff95e4-596b-420f-8576-a47ded494913 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:03.400294 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acff95e4-596b-420f-8576-a47ded494913 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:03.400577 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acff95e4-596b-420f-8576-a47ded494913 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:03.401255 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acff95e4-596b-420f-8576-a47ded494913 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:03.404917 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acff95e4-596b-420f-8576-a47ded494913 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:03.439963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 842/1711] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:03 2022] GET /identity/v3/roles => generated 1427 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:03.442791 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67faac1-9849-49d9-aa6b-f965c172a8a1 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:03.442924 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67faac1-9849-49d9-aa6b-f965c172a8a1 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:03.443397 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67faac1-9849-49d9-aa6b-f965c172a8a1 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:03.451551 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 843/1712] 10.222.0.102 () {54 vars in 917 bytes} [Wed Aug 24 15:47:03 2022] GET /identity => generated 271 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:47:03.452469 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67faac1-9849-49d9-aa6b-f965c172a8a1 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:03.455795 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-c67faac1-9849-49d9-aa6b-f965c172a8a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:03.458199 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a369ab7f-6b4e-4925-b478-8e8b66956d1e admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:03.464545 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220c653b-e258-437b-bb89-e1fe8d01c827 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:03.465344 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220c653b-e258-437b-bb89-e1fe8d01c827 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:03.465690 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220c653b-e258-437b-bb89-e1fe8d01c827 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:03.466104 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220c653b-e258-437b-bb89-e1fe8d01c827 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:03.472376 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-220c653b-e258-437b-bb89-e1fe8d01c827 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:03.478552 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67faac1-9849-49d9-aa6b-f965c172a8a1 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:03.483425 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85493330-5b49-4379-b51f-33a622125e34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:03.484114 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85493330-5b49-4379-b51f-33a622125e34 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:03.484600 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85493330-5b49-4379-b51f-33a622125e34 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:03.484880 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85493330-5b49-4379-b51f-33a622125e34 admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:03.485597 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85493330-5b49-4379-b51f-33a622125e34 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:47:03.485900 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85493330-5b49-4379-b51f-33a622125e34 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:03.487788 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85493330-5b49-4379-b51f-33a622125e34 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:03.489093 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220c653b-e258-437b-bb89-e1fe8d01c827 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:03.490969 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220c653b-e258-437b-bb89-e1fe8d01c827 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:03.514276 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 870/1713] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:03 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 255 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:03.516142 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de704c1c-ce52-474d-9e9e-225a0f588fc7 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:03.516441 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de704c1c-ce52-474d-9e9e-225a0f588fc7 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:03.516707 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 844/1714] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:47:03 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:47:03.516816 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de704c1c-ce52-474d-9e9e-225a0f588fc7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:03.517678 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85493330-5b49-4379-b51f-33a622125e34 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:03.522434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-85493330-5b49-4379-b51f-33a622125e34 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:03.566692 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-de704c1c-ce52-474d-9e9e-225a0f588fc7 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:03.567994 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-de704c1c-ce52-474d-9e9e-225a0f588fc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:03.603986 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 871/1715] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:03 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:03.604956 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de704c1c-ce52-474d-9e9e-225a0f588fc7 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:03.623865 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-de704c1c-ce52-474d-9e9e-225a0f588fc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:03.647480 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de704c1c-ce52-474d-9e9e-225a0f588fc7 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:03.673161 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22067c8-b07e-407e-8048-092c6db77923 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:03.673798 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22067c8-b07e-407e-8048-092c6db77923 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:03.674081 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22067c8-b07e-407e-8048-092c6db77923 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:03.674355 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22067c8-b07e-407e-8048-092c6db77923 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:03.675024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22067c8-b07e-407e-8048-092c6db77923 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:03.690293 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22067c8-b07e-407e-8048-092c6db77923 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:03.692858 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85493330-5b49-4379-b51f-33a622125e34 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:03.714631 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4aa162-9e03-4473-b7df-4c06d9186326 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:03.715262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4aa162-9e03-4473-b7df-4c06d9186326 service nova] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:03.715546 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4aa162-9e03-4473-b7df-4c06d9186326 service nova] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:03.715903 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4aa162-9e03-4473-b7df-4c06d9186326 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:03.724873 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 872/1716] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:03 2022] GET /identity/v3/roles => generated 1427 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:03.725642 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-dd4aa162-9e03-4473-b7df-4c06d9186326 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:03.726151 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e6f5d0-5f7d-477e-b11d-7141a7d3970d None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:03.727124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e6f5d0-5f7d-477e-b11d-7141a7d3970d None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:03.727891 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e6f5d0-5f7d-477e-b11d-7141a7d3970d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:03.768342 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-78e6f5d0-5f7d-477e-b11d-7141a7d3970d None None] MFA Rules not processed for user `48939674c1fd4cf599691d510fcd8858`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:03.769666 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-78e6f5d0-5f7d-477e-b11d-7141a7d3970d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:03.785989 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 873/1717] 10.222.0.102 () {58 vars in 1014 bytes} [Wed Aug 24 15:47:03 2022] POST /identity/v3/auth/tokens => generated 3131 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:03.787185 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e6f5d0-5f7d-477e-b11d-7141a7d3970d None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:03.819144 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e6f5d0-5f7d-477e-b11d-7141a7d3970d None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:03.844886 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9fc1a8-7a6e-4274-a488-6031822b0bcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:03.855484 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9fc1a8-7a6e-4274-a488-6031822b0bcc admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:03.855868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9fc1a8-7a6e-4274-a488-6031822b0bcc admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:03.858232 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9fc1a8-7a6e-4274-a488-6031822b0bcc admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:03.858970 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db9fc1a8-7a6e-4274-a488-6031822b0bcc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:03.865258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db9fc1a8-7a6e-4274-a488-6031822b0bcc admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:03.868642 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4aa162-9e03-4473-b7df-4c06d9186326 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:03.871686 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4aa162-9e03-4473-b7df-4c06d9186326 service nova] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:03.907749 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 845/1718] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:03 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 390 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:03.908898 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a415294-fc5f-4783-8d21-5d3c87f634c5 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:03.909923 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a415294-fc5f-4783-8d21-5d3c87f634c5 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:03.910217 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a415294-fc5f-4783-8d21-5d3c87f634c5 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:03.911666 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 846/1719] 10.222.0.102 () {54 vars in 966 bytes} [Wed Aug 24 15:47:03 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:47:03.912743 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6160fe5b-37f2-408a-94eb-4393a711c8b3 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:03.912976 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6160fe5b-37f2-408a-94eb-4393a711c8b3 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:03.913248 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6160fe5b-37f2-408a-94eb-4393a711c8b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:03.960760 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 874/1720] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:47:03 2022] POST /identity/v3/projects => generated 404 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:03.962497 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9fc1a8-7a6e-4274-a488-6031822b0bcc admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:03.976436 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-6160fe5b-37f2-408a-94eb-4393a711c8b3 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:03.978230 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-6160fe5b-37f2-408a-94eb-4393a711c8b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:04.130813 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 847/1721] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:03 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:04.132114 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6160fe5b-37f2-408a-94eb-4393a711c8b3 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:04.134724 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-6160fe5b-37f2-408a-94eb-4393a711c8b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:04.158784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9fc1a8-7a6e-4274-a488-6031822b0bcc admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:04.170676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:04.203145 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883cc51c-e288-4017-80d0-64989b92e269 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:04.203884 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883cc51c-e288-4017-80d0-64989b92e269 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:04.204169 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883cc51c-e288-4017-80d0-64989b92e269 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:04.204445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883cc51c-e288-4017-80d0-64989b92e269 admin admin] PATH_INFO: `/v3/projects/f0f68ee2560e449b8d77c3eeff196e53/users/cef7ea25a97248df9ae51cbd9980f1b7/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:04.239893 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883cc51c-e288-4017-80d0-64989b92e269 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f0f68ee2560e449b8d77c3eeff196e53, user_id=cef7ea25a97248df9ae51cbd9980f1b7, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:04.250549 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883cc51c-e288-4017-80d0-64989b92e269 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:04.290520 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 875/1722] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:47:03 2022] PUT /identity/v3/projects/f0f68ee2560e449b8d77c3eeff196e53/users/cef7ea25a97248df9ae51cbd9980f1b7/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 319 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:04.290520 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0f8725-74fa-4d4f-abcf-9fabdf6c0967 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:04.290520 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0f8725-74fa-4d4f-abcf-9fabdf6c0967 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:04.290520 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0f8725-74fa-4d4f-abcf-9fabdf6c0967 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:04.326251 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6160fe5b-37f2-408a-94eb-4393a711c8b3 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:04.332152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-2f0f8725-74fa-4d4f-abcf-9fabdf6c0967 None None] MFA Rules not processed for user `0ee0002a76c3471f819b37fef510ef99`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:04.337611 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-2f0f8725-74fa-4d4f-abcf-9fabdf6c0967 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:04.438538 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:04.444836 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32db519d-7481-4da3-8a79-500756a0b417 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:04.445501 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32db519d-7481-4da3-8a79-500756a0b417 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:04.445501 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32db519d-7481-4da3-8a79-500756a0b417 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:04.446272 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32db519d-7481-4da3-8a79-500756a0b417 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:04.461621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32db519d-7481-4da3-8a79-500756a0b417 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:04.463316 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32db519d-7481-4da3-8a79-500756a0b417 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:04.474993 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 876/1723] 10.222.0.102 () {58 vars in 1063 bytes} [Wed Aug 24 15:47:04 2022] POST /identity/v3/auth/tokens => generated 3318 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:04.476470 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0f8725-74fa-4d4f-abcf-9fabdf6c0967 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:04.485858 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 848/1724] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:04 2022] GET /identity/v3/auth/tokens => generated 3131 bytes in 354 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:04.487111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32db519d-7481-4da3-8a79-500756a0b417 service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:04.588446 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32db519d-7481-4da3-8a79-500756a0b417 service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:04.595365 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a303dcb-7efc-46da-84ac-8e20a7289eb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:04.596041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a303dcb-7efc-46da-84ac-8e20a7289eb2 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:04.596326 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a303dcb-7efc-46da-84ac-8e20a7289eb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:04.596603 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a303dcb-7efc-46da-84ac-8e20a7289eb2 admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:04.597576 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a303dcb-7efc-46da-84ac-8e20a7289eb2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:47:04.597995 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a303dcb-7efc-46da-84ac-8e20a7289eb2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:04.599702 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a303dcb-7efc-46da-84ac-8e20a7289eb2 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:04.613482 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 849/1725] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:47:04 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 127 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:47:04.614510 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a303dcb-7efc-46da-84ac-8e20a7289eb2 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:04.646096 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0f8725-74fa-4d4f-abcf-9fabdf6c0967 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:04.656424 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea0c8587-5388-4492-835c-b7130b3526fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:04.657306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0c8587-5388-4492-835c-b7130b3526fb admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:04.658471 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0c8587-5388-4492-835c-b7130b3526fb admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:04.660033 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0c8587-5388-4492-835c-b7130b3526fb admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:04.660800 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea0c8587-5388-4492-835c-b7130b3526fb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:04.667654 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea0c8587-5388-4492-835c-b7130b3526fb admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:04.679083 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:04.701558 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a303dcb-7efc-46da-84ac-8e20a7289eb2 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:04.712745 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a25eb83-a708-46d1-bc2f-ee97bb2c6034 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:04.713674 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a25eb83-a708-46d1-bc2f-ee97bb2c6034 service nova] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:04.713674 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a25eb83-a708-46d1-bc2f-ee97bb2c6034 service nova] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:04.713957 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a25eb83-a708-46d1-bc2f-ee97bb2c6034 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:04.715253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 877/1726] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:47:04 2022] POST /identity/v3/users => generated 303 bytes in 240 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:04.716392 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea0c8587-5388-4492-835c-b7130b3526fb admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:04.718464 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-2a25eb83-a708-46d1-bc2f-ee97bb2c6034 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:04.720928 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-ea0c8587-5388-4492-835c-b7130b3526fb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:04.749686 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea0c8587-5388-4492-835c-b7130b3526fb admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:04.764398 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be522ed-b560-4734-91a5-62467b0dfba6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:04.765059 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be522ed-b560-4734-91a5-62467b0dfba6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:04.765343 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be522ed-b560-4734-91a5-62467b0dfba6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:04.765845 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be522ed-b560-4734-91a5-62467b0dfba6 admin admin] PATH_INFO: `/v3/projects/d20a286c27cf4c8cbfbaffe60c2ad75f` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:04.767246 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be522ed-b560-4734-91a5-62467b0dfba6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d20a286c27cf4c8cbfbaffe60c2ad75f)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:04.773674 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be522ed-b560-4734-91a5-62467b0dfba6 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:04.865935 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a25eb83-a708-46d1-bc2f-ee97bb2c6034 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:04.868127 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a25eb83-a708-46d1-bc2f-ee97bb2c6034 service nova] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:04.880441 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4be522ed-b560-4734-91a5-62467b0dfba6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd20a286c27cf4c8cbfbaffe60c2ad75f', 'request_id': 'req-4be522ed-b560-4734-91a5-62467b0dfba6'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:04.888888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4be522ed-b560-4734-91a5-62467b0dfba6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd20a286c27cf4c8cbfbaffe60c2ad75f', 'request_id': 'req-4be522ed-b560-4734-91a5-62467b0dfba6'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:04.893681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 850/1727] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:04 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 279 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:04.895663 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4be522ed-b560-4734-91a5-62467b0dfba6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd20a286c27cf4c8cbfbaffe60c2ad75f', 'request_id': 'req-4be522ed-b560-4734-91a5-62467b0dfba6'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:04.899847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af68211e-65a5-4828-a0c1-31d489bec760 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:04.900498 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af68211e-65a5-4828-a0c1-31d489bec760 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:04.901222 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af68211e-65a5-4828-a0c1-31d489bec760 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:04.917729 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4be522ed-b560-4734-91a5-62467b0dfba6 admin admin] The token cache is being invalidate because project d20a286c27cf4c8cbfbaffe60c2ad75f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:47:04.918167 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4be522ed-b560-4734-91a5-62467b0dfba6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:04.919135 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4be522ed-b560-4734-91a5-62467b0dfba6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:04.923972 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 878/1728] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:47:04 2022] DELETE /identity/v3/projects/d20a286c27cf4c8cbfbaffe60c2ad75f => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:04.923972 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be522ed-b560-4734-91a5-62467b0dfba6 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:04.927259 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-4be522ed-b560-4734-91a5-62467b0dfba6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:04.986701 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-af68211e-65a5-4828-a0c1-31d489bec760 None None] MFA Rules not processed for user `cef7ea25a97248df9ae51cbd9980f1b7`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:04.989022 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-af68211e-65a5-4828-a0c1-31d489bec760 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:05.201795 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be522ed-b560-4734-91a5-62467b0dfba6 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:05.220591 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 851/1729] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:04 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 325 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:05.221926 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af68211e-65a5-4828-a0c1-31d489bec760 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:05.224609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b05634f-45ba-43ef-89fa-f3cd59a0d0c8 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0ee0002a76c3471f819b37fef510ef99', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:05.226151 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-af68211e-65a5-4828-a0c1-31d489bec760 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:05.238623 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b05634f-45ba-43ef-89fa-f3cd59a0d0c8 service placement] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:05.239298 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b05634f-45ba-43ef-89fa-f3cd59a0d0c8 service placement] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:05.240104 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b05634f-45ba-43ef-89fa-f3cd59a0d0c8 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:05.246806 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-8b05634f-45ba-43ef-89fa-f3cd59a0d0c8 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:05.287665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b05634f-45ba-43ef-89fa-f3cd59a0d0c8 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:05.289776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b05634f-45ba-43ef-89fa-f3cd59a0d0c8 service placement] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:05.311856 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:05.332060 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 879/1730] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:04 2022] GET /identity/v3/auth/tokens => generated 3318 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:05.333283 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b05634f-45ba-43ef-89fa-f3cd59a0d0c8 service placement] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:05.338140 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-8b05634f-45ba-43ef-89fa-f3cd59a0d0c8 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:05.500294 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af68211e-65a5-4828-a0c1-31d489bec760 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:05.516798 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d34460-afbf-4dc9-ad74-3e08ac1c8bbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:05.517691 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d34460-afbf-4dc9-ad74-3e08ac1c8bbe admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:05.517979 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d34460-afbf-4dc9-ad74-3e08ac1c8bbe admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:05.518259 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d34460-afbf-4dc9-ad74-3e08ac1c8bbe admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:05.519060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d34460-afbf-4dc9-ad74-3e08ac1c8bbe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:05.520882 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d34460-afbf-4dc9-ad74-3e08ac1c8bbe admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:05.548415 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b05634f-45ba-43ef-89fa-f3cd59a0d0c8 service placement] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:05.563595 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2619398f-2624-472e-bc2a-a78d1e3c522d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:05.572132 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2619398f-2624-472e-bc2a-a78d1e3c522d admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:05.574922 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2619398f-2624-472e-bc2a-a78d1e3c522d admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:05.577259 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2619398f-2624-472e-bc2a-a78d1e3c522d admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:05.579883 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2619398f-2624-472e-bc2a-a78d1e3c522d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:05.585042 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2619398f-2624-472e-bc2a-a78d1e3c522d admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:05.654417 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 852/1731] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:47:05 2022] POST /identity/v3/projects => generated 374 bytes in 429 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:05.654417 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d34460-afbf-4dc9-ad74-3e08ac1c8bbe admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:05.655764 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 880/1732] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:05 2022] GET /identity/v3/roles => generated 1427 bytes in 323 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:05.656992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-24d34460-afbf-4dc9-ad74-3e08ac1c8bbe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:05.670399 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2619398f-2624-472e-bc2a-a78d1e3c522d admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:05.691036 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-2619398f-2624-472e-bc2a-a78d1e3c522d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:05.700478 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:05.740919 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2619398f-2624-472e-bc2a-a78d1e3c522d admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:05.897781 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d34460-afbf-4dc9-ad74-3e08ac1c8bbe admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:05.918879 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d9e9e4-63a4-498a-9b73-3bd72d8b8ff1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:05.919524 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d9e9e4-63a4-498a-9b73-3bd72d8b8ff1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:05.920878 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d9e9e4-63a4-498a-9b73-3bd72d8b8ff1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:05.923698 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d9e9e4-63a4-498a-9b73-3bd72d8b8ff1 admin admin] PATH_INFO: `/v3/users/de56a2b58a8146aebb18ade78a98ea82` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:05.925334 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d9e9e4-63a4-498a-9b73-3bd72d8b8ff1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=de56a2b58a8146aebb18ade78a98ea82)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:05.927293 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d9e9e4-63a4-498a-9b73-3bd72d8b8ff1 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:05.954125 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0790c5-7e60-4514-9b0c-d1b8360142ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:05.955013 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0790c5-7e60-4514-9b0c-d1b8360142ba admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:05.955157 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0790c5-7e60-4514-9b0c-d1b8360142ba admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:05.955620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0790c5-7e60-4514-9b0c-d1b8360142ba admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:05.956008 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0790c5-7e60-4514-9b0c-d1b8360142ba admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:05.957947 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0790c5-7e60-4514-9b0c-d1b8360142ba admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:06.083233 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 881/1733] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:47:05 2022] POST /identity/v3/projects => generated 376 bytes in 414 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:06.089297 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b0066c-650d-4724-baf0-eeb33468381a None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:06.089841 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b0066c-650d-4724-baf0-eeb33468381a None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:06.090279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b0066c-650d-4724-baf0-eeb33468381a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:06.121097 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-60d9e9e4-63a4-498a-9b73-3bd72d8b8ff1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'de56a2b58a8146aebb18ade78a98ea82'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:06.124077 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-60d9e9e4-63a4-498a-9b73-3bd72d8b8ff1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'de56a2b58a8146aebb18ade78a98ea82'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:06.146213 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-60d9e9e4-63a4-498a-9b73-3bd72d8b8ff1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'de56a2b58a8146aebb18ade78a98ea82'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:06.170681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-60d9e9e4-63a4-498a-9b73-3bd72d8b8ff1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'de56a2b58a8146aebb18ade78a98ea82'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:06.171961 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-60d9e9e4-63a4-498a-9b73-3bd72d8b8ff1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'de56a2b58a8146aebb18ade78a98ea82'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:06.184492 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-b9b0066c-650d-4724-baf0-eeb33468381a None None] MFA Rules not processed for user `2fe5724620da46ddbefbdfd442dc8d73`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:06.191808 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-b9b0066c-650d-4724-baf0-eeb33468381a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:06.216039 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 853/1734] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:47:05 2022] DELETE /identity/v3/users/de56a2b58a8146aebb18ade78a98ea82 => generated 0 bytes in 565 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:06.218186 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d9e9e4-63a4-498a-9b73-3bd72d8b8ff1 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:06.220465 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-60d9e9e4-63a4-498a-9b73-3bd72d8b8ff1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:06.389308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 882/1735] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:06 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 305 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:06.390391 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b0066c-650d-4724-baf0-eeb33468381a None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:06.397327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-b9b0066c-650d-4724-baf0-eeb33468381a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:06.413598 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d9e9e4-63a4-498a-9b73-3bd72d8b8ff1 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:06.438811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ff63e1-2e0f-4beb-8c02-6f19a64ddb7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:06.440009 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ff63e1-2e0f-4beb-8c02-6f19a64ddb7b admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:06.441933 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ff63e1-2e0f-4beb-8c02-6f19a64ddb7b admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:06.442593 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ff63e1-2e0f-4beb-8c02-6f19a64ddb7b admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:06.443416 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ff63e1-2e0f-4beb-8c02-6f19a64ddb7b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:06.448804 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ff63e1-2e0f-4beb-8c02-6f19a64ddb7b admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:06.479413 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:06.540798 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 854/1736] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:47:06 2022] POST /identity/v3/users => generated 288 bytes in 323 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:06.542216 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ff63e1-2e0f-4beb-8c02-6f19a64ddb7b admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:06.547673 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-43ff63e1-2e0f-4beb-8c02-6f19a64ddb7b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:06.591898 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ff63e1-2e0f-4beb-8c02-6f19a64ddb7b admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:06.598837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c6d1bc-f991-4395-81e9-0957ed999cfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:06.599934 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c6d1bc-f991-4395-81e9-0957ed999cfe admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:06.600371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c6d1bc-f991-4395-81e9-0957ed999cfe admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:06.600985 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c6d1bc-f991-4395-81e9-0957ed999cfe admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:06.602373 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c6d1bc-f991-4395-81e9-0957ed999cfe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:06.606049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c6d1bc-f991-4395-81e9-0957ed999cfe admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:06.636346 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=83,keystone:DELETE=10,keystone:INSERT=7 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:06.668863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b0066c-650d-4724-baf0-eeb33468381a None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:06.684950 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 855/1737] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:47:06 2022] POST /identity/v3/users => generated 295 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:06.685895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c6d1bc-f991-4395-81e9-0957ed999cfe admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:06.704850 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c40da1-e8bf-485c-b00f-fbfacd6649f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:06.707700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c40da1-e8bf-485c-b00f-fbfacd6649f6 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:06.708199 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c40da1-e8bf-485c-b00f-fbfacd6649f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:06.710508 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c40da1-e8bf-485c-b00f-fbfacd6649f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:06.712109 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c40da1-e8bf-485c-b00f-fbfacd6649f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:06.717863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c40da1-e8bf-485c-b00f-fbfacd6649f6 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:06.767462 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c6d1bc-f991-4395-81e9-0957ed999cfe admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:06.782251 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 883/1738] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:06 2022] GET /identity/v3/roles => generated 1427 bytes in 391 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:06.787311 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c40da1-e8bf-485c-b00f-fbfacd6649f6 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:06.795160 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d5a83b-9515-4094-8bd8-67691f021356 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:06.796160 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d5a83b-9515-4094-8bd8-67691f021356 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:06.796629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d5a83b-9515-4094-8bd8-67691f021356 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:06.800594 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d5a83b-9515-4094-8bd8-67691f021356 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:06.801780 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d5a83b-9515-4094-8bd8-67691f021356 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:06.804358 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d5a83b-9515-4094-8bd8-67691f021356 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:06.844839 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c40da1-e8bf-485c-b00f-fbfacd6649f6 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:06.852037 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 856/1739] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:06 2022] GET /identity/v3/roles => generated 1427 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:06.853344 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d5a83b-9515-4094-8bd8-67691f021356 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:06.882201 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d5a83b-9515-4094-8bd8-67691f021356 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:06.886546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5934ffdc-b2e9-46d0-92dc-dab05fa9d203 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:06.891668 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5934ffdc-b2e9-46d0-92dc-dab05fa9d203 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:06.892872 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5934ffdc-b2e9-46d0-92dc-dab05fa9d203 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:06.894504 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5934ffdc-b2e9-46d0-92dc-dab05fa9d203 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:06.895739 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5934ffdc-b2e9-46d0-92dc-dab05fa9d203 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:06.904623 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ad12b9-3456-47b3-86e3-e6a14c0801a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:06.908658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ad12b9-3456-47b3-86e3-e6a14c0801a1 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:06.910125 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ad12b9-3456-47b3-86e3-e6a14c0801a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:06.910125 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ad12b9-3456-47b3-86e3-e6a14c0801a1 admin admin] PATH_INFO: `/v3/projects/f78537131d144cbebdf40cd763440080/users/7999201deb0d4f0391df21fdd0cdb60b/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:06.926652 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5934ffdc-b2e9-46d0-92dc-dab05fa9d203 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:06.975061 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ad12b9-3456-47b3-86e3-e6a14c0801a1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f78537131d144cbebdf40cd763440080, user_id=7999201deb0d4f0391df21fdd0cdb60b, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:06.982047 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ad12b9-3456-47b3-86e3-e6a14c0801a1 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:06.997704 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 884/1740] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:06 2022] GET /identity/v3/roles => generated 1427 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:07.003020 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5934ffdc-b2e9-46d0-92dc-dab05fa9d203 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:07.036820 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 857/1741] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:47:06 2022] PUT /identity/v3/projects/f78537131d144cbebdf40cd763440080/users/7999201deb0d4f0391df21fdd0cdb60b/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:07.040701 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ad12b9-3456-47b3-86e3-e6a14c0801a1 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:07.046639 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5934ffdc-b2e9-46d0-92dc-dab05fa9d203 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:07.267728 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934eca2e-9c1f-4d1b-b2ed-6bc4879c56e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:07.268867 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934eca2e-9c1f-4d1b-b2ed-6bc4879c56e5 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:07.269347 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934eca2e-9c1f-4d1b-b2ed-6bc4879c56e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:07.269810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934eca2e-9c1f-4d1b-b2ed-6bc4879c56e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:07.271405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-934eca2e-9c1f-4d1b-b2ed-6bc4879c56e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:07.273275 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-934eca2e-9c1f-4d1b-b2ed-6bc4879c56e5 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:07.282674 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ad12b9-3456-47b3-86e3-e6a14c0801a1 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:07.299746 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 885/1742] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:06 2022] GET /identity/v3/roles => generated 1427 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:07.309441 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada0ffb7-98dd-47fc-9ada-c99b139a1980 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:07.311486 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730152fc-ff5e-4dbb-9d8a-3d17443c046e None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:07.312070 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada0ffb7-98dd-47fc-9ada-c99b139a1980 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:07.314307 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada0ffb7-98dd-47fc-9ada-c99b139a1980 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:07.314917 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730152fc-ff5e-4dbb-9d8a-3d17443c046e None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:07.315655 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada0ffb7-98dd-47fc-9ada-c99b139a1980 admin admin] PATH_INFO: `/v3/projects/7c1409f1315c43dbb6c4080fc3ed5ca0/users/5e54c3c38a5444dcb8b377410e88b101/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:07.317456 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730152fc-ff5e-4dbb-9d8a-3d17443c046e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:07.352805 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada0ffb7-98dd-47fc-9ada-c99b139a1980 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7c1409f1315c43dbb6c4080fc3ed5ca0, user_id=5e54c3c38a5444dcb8b377410e88b101, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:07.361356 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada0ffb7-98dd-47fc-9ada-c99b139a1980 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:07.363422 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-730152fc-ff5e-4dbb-9d8a-3d17443c046e None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:07.370426 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-730152fc-ff5e-4dbb-9d8a-3d17443c046e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:07.394999 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 886/1743] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:07 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:07.395408 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 858/1744] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:47:07 2022] PUT /identity/v3/projects/7c1409f1315c43dbb6c4080fc3ed5ca0/users/5e54c3c38a5444dcb8b377410e88b101/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 355 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:07.396338 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730152fc-ff5e-4dbb-9d8a-3d17443c046e None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:07.409472 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada0ffb7-98dd-47fc-9ada-c99b139a1980 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:07.429061 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730152fc-ff5e-4dbb-9d8a-3d17443c046e None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:07.559261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada0ffb7-98dd-47fc-9ada-c99b139a1980 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:07.572115 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c119f3-d40f-437c-a581-31fed05e08ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:07.572983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c119f3-d40f-437c-a581-31fed05e08ec admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:07.573265 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c119f3-d40f-437c-a581-31fed05e08ec admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:07.575662 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c119f3-d40f-437c-a581-31fed05e08ec admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:07.578419 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c119f3-d40f-437c-a581-31fed05e08ec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:07.581540 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c119f3-d40f-437c-a581-31fed05e08ec admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:07.596538 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482c2a9d-b51d-4877-9870-dfceb8b1feb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:07.596809 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482c2a9d-b51d-4877-9870-dfceb8b1feb1 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:07.599080 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482c2a9d-b51d-4877-9870-dfceb8b1feb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:07.599425 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482c2a9d-b51d-4877-9870-dfceb8b1feb1 admin admin] PATH_INFO: `/v3/projects/ce7dce2a40e945f6be112ac8733b8b5f/users/202224f89f274d398b34f9653bd769cc/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:07.622182 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 859/1745] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:07 2022] GET /identity/v3/roles => generated 1427 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 15:47:07.623156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c119f3-d40f-437c-a581-31fed05e08ec admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:07.628241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-36c119f3-d40f-437c-a581-31fed05e08ec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:07.647277 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-482c2a9d-b51d-4877-9870-dfceb8b1feb1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ce7dce2a40e945f6be112ac8733b8b5f, user_id=202224f89f274d398b34f9653bd769cc, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:07.650545 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-482c2a9d-b51d-4877-9870-dfceb8b1feb1 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:07.693319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 887/1746] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:47:07 2022] PUT /identity/v3/projects/ce7dce2a40e945f6be112ac8733b8b5f/users/202224f89f274d398b34f9653bd769cc/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:07.694394 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482c2a9d-b51d-4877-9870-dfceb8b1feb1 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:07.776797 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:07.781805 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:07.825260 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c119f3-d40f-437c-a581-31fed05e08ec admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:07.899924 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482c2a9d-b51d-4877-9870-dfceb8b1feb1 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:07.918057 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce4e1e2-fd18-4a8e-9755-40c56ca5e97b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:07.919352 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce4e1e2-fd18-4a8e-9755-40c56ca5e97b admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:07.920988 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce4e1e2-fd18-4a8e-9755-40c56ca5e97b admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:07.922552 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce4e1e2-fd18-4a8e-9755-40c56ca5e97b admin admin] PATH_INFO: `/v3/projects/7c1409f1315c43dbb6c4080fc3ed5ca0/users/5e54c3c38a5444dcb8b377410e88b101/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:07.931509 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce4e1e2-fd18-4a8e-9755-40c56ca5e97b admin admin] RBAC: Authorizing `identity:create_grant(project_id=7c1409f1315c43dbb6c4080fc3ed5ca0, user_id=5e54c3c38a5444dcb8b377410e88b101, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:07.934273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce4e1e2-fd18-4a8e-9755-40c56ca5e97b admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:07.971405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 888/1747] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:47:07 2022] PUT /identity/v3/projects/7c1409f1315c43dbb6c4080fc3ed5ca0/users/5e54c3c38a5444dcb8b377410e88b101/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:07.975339 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e09725-3db0-43aa-b706-eb22240b0631 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:07.975981 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e09725-3db0-43aa-b706-eb22240b0631 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:07.977466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e09725-3db0-43aa-b706-eb22240b0631 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:07.983102 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1af8f09-b61f-4d4f-9756-f2bbf2a1422e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:07.983758 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1af8f09-b61f-4d4f-9756-f2bbf2a1422e service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:07.984041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1af8f09-b61f-4d4f-9756-f2bbf2a1422e service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:07.984317 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1af8f09-b61f-4d4f-9756-f2bbf2a1422e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:08.004696 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1af8f09-b61f-4d4f-9756-f2bbf2a1422e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:08.006157 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1af8f09-b61f-4d4f-9756-f2bbf2a1422e service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:08.025376 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-35e09725-3db0-43aa-b706-eb22240b0631 None None] MFA Rules not processed for user `202224f89f274d398b34f9653bd769cc`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:08.027927 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-35e09725-3db0-43aa-b706-eb22240b0631 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:08.032963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 860/1748] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:07 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:08.035094 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1af8f09-b61f-4d4f-9756-f2bbf2a1422e service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:08.040985 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-f1af8f09-b61f-4d4f-9756-f2bbf2a1422e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:08.057545 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1af8f09-b61f-4d4f-9756-f2bbf2a1422e service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:08.141863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 889/1749] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:07 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:08.144082 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8feb8d1-3e3e-40b0-b0ec-24855fb99081 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:08.144431 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8feb8d1-3e3e-40b0-b0ec-24855fb99081 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:08.144816 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8feb8d1-3e3e-40b0-b0ec-24855fb99081 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:08.168447 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e20599-0c05-4404-9d3a-2478434c7ba2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:08.169415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e20599-0c05-4404-9d3a-2478434c7ba2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:08.169896 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e20599-0c05-4404-9d3a-2478434c7ba2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:08.170438 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e20599-0c05-4404-9d3a-2478434c7ba2 admin admin] PATH_INFO: `/v3/projects/e1dc926c14aa40d18e0cf91c1e417bde` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:08.172019 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e20599-0c05-4404-9d3a-2478434c7ba2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e1dc926c14aa40d18e0cf91c1e417bde)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:08.174582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e20599-0c05-4404-9d3a-2478434c7ba2 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:08.231191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-b8feb8d1-3e3e-40b0-b0ec-24855fb99081 None None] MFA Rules not processed for user `5e54c3c38a5444dcb8b377410e88b101`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:08.235548 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-b8feb8d1-3e3e-40b0-b0ec-24855fb99081 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:08.286296 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d7e20599-0c05-4404-9d3a-2478434c7ba2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e1dc926c14aa40d18e0cf91c1e417bde', 'request_id': 'req-d7e20599-0c05-4404-9d3a-2478434c7ba2'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:08.303793 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d7e20599-0c05-4404-9d3a-2478434c7ba2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e1dc926c14aa40d18e0cf91c1e417bde', 'request_id': 'req-d7e20599-0c05-4404-9d3a-2478434c7ba2'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:08.321446 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d7e20599-0c05-4404-9d3a-2478434c7ba2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e1dc926c14aa40d18e0cf91c1e417bde', 'request_id': 'req-d7e20599-0c05-4404-9d3a-2478434c7ba2'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:08.350761 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d7e20599-0c05-4404-9d3a-2478434c7ba2 admin admin] The token cache is being invalidate because project e1dc926c14aa40d18e0cf91c1e417bde was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:47:08.354374 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d7e20599-0c05-4404-9d3a-2478434c7ba2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:08.356069 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d7e20599-0c05-4404-9d3a-2478434c7ba2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:08.361212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 861/1750] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:47:08 2022] DELETE /identity/v3/projects/e1dc926c14aa40d18e0cf91c1e417bde => generated 0 bytes in 327 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:08.363403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e20599-0c05-4404-9d3a-2478434c7ba2 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:08.370656 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-d7e20599-0c05-4404-9d3a-2478434c7ba2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:08.407689 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 890/1751] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:08 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 263 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:08.408346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5d6791-3458-40a1-b293-aacdfe15672a None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:08.408842 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5d6791-3458-40a1-b293-aacdfe15672a None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:08.409322 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5d6791-3458-40a1-b293-aacdfe15672a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:08.457881 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-fb5d6791-3458-40a1-b293-aacdfe15672a None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:08.459106 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-fb5d6791-3458-40a1-b293-aacdfe15672a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:08.463000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:08.513916 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:08.572556 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e20599-0c05-4404-9d3a-2478434c7ba2 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:08.590500 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219795c0-f34e-49fb-8f5e-a976b0ed27a8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:08.591135 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219795c0-f34e-49fb-8f5e-a976b0ed27a8 service nova] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:08.591417 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219795c0-f34e-49fb-8f5e-a976b0ed27a8 service nova] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:08.591705 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219795c0-f34e-49fb-8f5e-a976b0ed27a8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:08.594818 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 891/1752] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:08 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:08.595846 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5d6791-3458-40a1-b293-aacdfe15672a None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:08.599611 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-219795c0-f34e-49fb-8f5e-a976b0ed27a8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:08.602381 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-fb5d6791-3458-40a1-b293-aacdfe15672a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:08.761969 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5d6791-3458-40a1-b293-aacdfe15672a None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:08.772217 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-219795c0-f34e-49fb-8f5e-a976b0ed27a8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:08.774831 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-219795c0-f34e-49fb-8f5e-a976b0ed27a8 service nova] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:08.777443 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a4f3ee-e9a0-45d7-9236-c4a4f95a1264 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:08.778175 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a4f3ee-e9a0-45d7-9236-c4a4f95a1264 service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:08.778410 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a4f3ee-e9a0-45d7-9236-c4a4f95a1264 service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:08.778687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a4f3ee-e9a0-45d7-9236-c4a4f95a1264 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:08.824311 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 862/1753] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:08 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 450 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:08.826073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b4e472-865e-4102-90cf-1e118ff0a186 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:08.826630 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b4e472-865e-4102-90cf-1e118ff0a186 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:08.827002 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b4e472-865e-4102-90cf-1e118ff0a186 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:08.903361 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-70b4e472-865e-4102-90cf-1e118ff0a186 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:08.909095 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-70b4e472-865e-4102-90cf-1e118ff0a186 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:08.922723 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 863/1754] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:08 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:08.932853 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b4e472-865e-4102-90cf-1e118ff0a186 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:08.940830 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a4f3ee-e9a0-45d7-9236-c4a4f95a1264 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:08.943080 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a4f3ee-e9a0-45d7-9236-c4a4f95a1264 service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:08.957483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b4e472-865e-4102-90cf-1e118ff0a186 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:08.984698 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 892/1755] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:08 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:08.985233 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a57914-45cc-4df1-b905-6836e6119c0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:08.987108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a57914-45cc-4df1-b905-6836e6119c0e admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:08.990332 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a57914-45cc-4df1-b905-6836e6119c0e admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:08.990727 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a57914-45cc-4df1-b905-6836e6119c0e admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:08.991518 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a57914-45cc-4df1-b905-6836e6119c0e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:47:08.991955 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a57914-45cc-4df1-b905-6836e6119c0e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:08.996042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a57914-45cc-4df1-b905-6836e6119c0e admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:09.019929 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 864/1756] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:47:08 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:47:09.028096 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a4f3ee-e9a0-45d7-9236-c4a4f95a1264 service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:09.053340 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a57914-45cc-4df1-b905-6836e6119c0e admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:09.061634 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-e7a57914-45cc-4df1-b905-6836e6119c0e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:09.068580 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a4f3ee-e9a0-45d7-9236-c4a4f95a1264 service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:09.091998 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f94e78-5482-40b7-be39-cb51482225e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:09.094718 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f94e78-5482-40b7-be39-cb51482225e4 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:09.096804 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f94e78-5482-40b7-be39-cb51482225e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:09.098780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f94e78-5482-40b7-be39-cb51482225e4 admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:09.099673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f94e78-5482-40b7-be39-cb51482225e4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:09.108373 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f94e78-5482-40b7-be39-cb51482225e4 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:09.113667 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a57914-45cc-4df1-b905-6836e6119c0e admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:09.130627 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec40d5a-0be5-4bc0-9136-9179634cdcc7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:09.132261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec40d5a-0be5-4bc0-9136-9179634cdcc7 service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:09.135023 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec40d5a-0be5-4bc0-9136-9179634cdcc7 service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:09.135418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec40d5a-0be5-4bc0-9136-9179634cdcc7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:09.142582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-eec40d5a-0be5-4bc0-9136-9179634cdcc7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:09.183322 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 893/1757] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:47:09 2022] POST /identity/v3/projects => generated 386 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:09.184294 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f94e78-5482-40b7-be39-cb51482225e4 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:09.310955 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec40d5a-0be5-4bc0-9136-9179634cdcc7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:09.312486 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec40d5a-0be5-4bc0-9136-9179634cdcc7 service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:09.324801 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f94e78-5482-40b7-be39-cb51482225e4 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:09.325335 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 865/1758] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:09 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 276 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:09.343577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a53774f-cae3-4e1e-a99d-b399a77bae70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:09.344245 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a53774f-cae3-4e1e-a99d-b399a77bae70 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:09.344581 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a53774f-cae3-4e1e-a99d-b399a77bae70 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:09.344794 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a53774f-cae3-4e1e-a99d-b399a77bae70 admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:09.345440 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a53774f-cae3-4e1e-a99d-b399a77bae70 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:09.347404 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a53774f-cae3-4e1e-a99d-b399a77bae70 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:09.365854 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:09.415668 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 894/1759] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:47:09 2022] POST /identity/v3/users => generated 294 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:09.419742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec40d5a-0be5-4bc0-9136-9179634cdcc7 service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:09.435729 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec40d5a-0be5-4bc0-9136-9179634cdcc7 service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:09.450477 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a632894c-ca06-4f0d-91b4-f4a60ce43b8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:09.465112 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabb25d5-537b-48ae-98d6-d1f7a1611f6e None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:09.471874 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabb25d5-537b-48ae-98d6-d1f7a1611f6e None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:09.472310 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabb25d5-537b-48ae-98d6-d1f7a1611f6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:09.487500 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a632894c-ca06-4f0d-91b4-f4a60ce43b8d admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:09.488039 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a632894c-ca06-4f0d-91b4-f4a60ce43b8d admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:09.488418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a632894c-ca06-4f0d-91b4-f4a60ce43b8d admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:09.490081 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a632894c-ca06-4f0d-91b4-f4a60ce43b8d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:09.494147 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a632894c-ca06-4f0d-91b4-f4a60ce43b8d admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:09.544509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 866/1760] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:09 2022] GET /identity/v3/roles => generated 1427 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:09.548197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a632894c-ca06-4f0d-91b4-f4a60ce43b8d admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:09.582031 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a632894c-ca06-4f0d-91b4-f4a60ce43b8d admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:09.595183 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c63e9d8-235e-46a8-ae2a-dfaa169cf231 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:09.596434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c63e9d8-235e-46a8-ae2a-dfaa169cf231 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:09.597742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c63e9d8-235e-46a8-ae2a-dfaa169cf231 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:09.598128 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c63e9d8-235e-46a8-ae2a-dfaa169cf231 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:09.600593 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c63e9d8-235e-46a8-ae2a-dfaa169cf231 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:09.609491 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c63e9d8-235e-46a8-ae2a-dfaa169cf231 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:09.611894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-dabb25d5-537b-48ae-98d6-d1f7a1611f6e None None] MFA Rules not processed for user `a6c17620ead14cb49d962990e1b1e0d8`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:09.613606 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-dabb25d5-537b-48ae-98d6-d1f7a1611f6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:09.641803 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 867/1761] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:09 2022] GET /identity/v3/roles => generated 1427 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:09.644492 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c63e9d8-235e-46a8-ae2a-dfaa169cf231 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:09.676864 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c63e9d8-235e-46a8-ae2a-dfaa169cf231 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:09.691938 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86312f78-87d4-484e-8354-3104ac722400 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:09.697796 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86312f78-87d4-484e-8354-3104ac722400 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:09.698079 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86312f78-87d4-484e-8354-3104ac722400 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:09.698405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86312f78-87d4-484e-8354-3104ac722400 admin admin] PATH_INFO: `/v3/projects/a71f2a925d0a4dc085ba518e5779b93d/users/fc61eb142e1a49558fa736f8406a3c7b/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:09.725004 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 895/1762] 10.222.0.102 () {58 vars in 1014 bytes} [Wed Aug 24 15:47:09 2022] POST /identity/v3/auth/tokens => generated 3205 bytes in 261 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:47:09.808805 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86312f78-87d4-484e-8354-3104ac722400 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a71f2a925d0a4dc085ba518e5779b93d, user_id=fc61eb142e1a49558fa736f8406a3c7b, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:09.813022 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86312f78-87d4-484e-8354-3104ac722400 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:09.893215 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:09.914540 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 868/1763] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:47:09 2022] PUT /identity/v3/projects/a71f2a925d0a4dc085ba518e5779b93d/users/fc61eb142e1a49558fa736f8406a3c7b/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:09.916222 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246df674-cf8e-4eeb-a98c-6ab0f8ca75d7 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:09.916694 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246df674-cf8e-4eeb-a98c-6ab0f8ca75d7 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:09.917113 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246df674-cf8e-4eeb-a98c-6ab0f8ca75d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:10.057947 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86312f78-87d4-484e-8354-3104ac722400 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:10.060524 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-86312f78-87d4-484e-8354-3104ac722400 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:10.105213 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-246df674-cf8e-4eeb-a98c-6ab0f8ca75d7 None None] MFA Rules not processed for user `fc61eb142e1a49558fa736f8406a3c7b`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:10.120426 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-246df674-cf8e-4eeb-a98c-6ab0f8ca75d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:10.374423 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 896/1764] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:09 2022] POST /identity/v3/auth/tokens => generated 3272 bytes in 457 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:47:10.376648 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246df674-cf8e-4eeb-a98c-6ab0f8ca75d7 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:10.416051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86312f78-87d4-484e-8354-3104ac722400 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:10.430925 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07152aff-083e-4c98-a989-78dc6b4273e8 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a070d4fd8c4466c857bd03965c11349', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:10.432061 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07152aff-083e-4c98-a989-78dc6b4273e8 service swift] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:10.432561 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07152aff-083e-4c98-a989-78dc6b4273e8 service swift] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:10.435948 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07152aff-083e-4c98-a989-78dc6b4273e8 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:10.598043 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:10.607027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07152aff-083e-4c98-a989-78dc6b4273e8 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:10.612463 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07152aff-083e-4c98-a989-78dc6b4273e8 service swift] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:10.628339 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246df674-cf8e-4eeb-a98c-6ab0f8ca75d7 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:10.635125 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 869/1765] 10.222.0.102 () {60 vars in 1389 bytes} [Wed Aug 24 15:47:10 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 577 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 24 15:47:10.643926 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07152aff-083e-4c98-a989-78dc6b4273e8 service swift] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:10.652274 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-07152aff-083e-4c98-a989-78dc6b4273e8 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:10.652729 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92067d1-606d-4c2b-bb5b-e1067cfec8a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:10.670174 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92067d1-606d-4c2b-bb5b-e1067cfec8a7 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:10.671394 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92067d1-606d-4c2b-bb5b-e1067cfec8a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:10.671722 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92067d1-606d-4c2b-bb5b-e1067cfec8a7 admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:10.692714 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92067d1-606d-4c2b-bb5b-e1067cfec8a7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:10.714169 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92067d1-606d-4c2b-bb5b-e1067cfec8a7 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:10.725687 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07152aff-083e-4c98-a989-78dc6b4273e8 service swift] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:10.741517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39861c29-bb5b-4783-a5dd-9003c003c60a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:10.744809 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39861c29-bb5b-4783-a5dd-9003c003c60a admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:10.745657 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39861c29-bb5b-4783-a5dd-9003c003c60a admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:10.746887 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39861c29-bb5b-4783-a5dd-9003c003c60a admin admin] PATH_INFO: `/v3/users/202224f89f274d398b34f9653bd769cc` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:10.751029 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39861c29-bb5b-4783-a5dd-9003c003c60a admin admin] RBAC: Authorizing `identity:delete_user(user_id=202224f89f274d398b34f9653bd769cc)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:10.753886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39861c29-bb5b-4783-a5dd-9003c003c60a admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:10.859548 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 897/1766] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:47:10 2022] POST /identity/v3/projects => generated 386 bytes in 483 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:10.863802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92067d1-606d-4c2b-bb5b-e1067cfec8a7 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:10.977815 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-39861c29-bb5b-4783-a5dd-9003c003c60a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '202224f89f274d398b34f9653bd769cc'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:10.979876 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-39861c29-bb5b-4783-a5dd-9003c003c60a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '202224f89f274d398b34f9653bd769cc'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:11.022453 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-39861c29-bb5b-4783-a5dd-9003c003c60a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '202224f89f274d398b34f9653bd769cc'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:11.078923 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92067d1-606d-4c2b-bb5b-e1067cfec8a7 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:11.084890 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-39861c29-bb5b-4783-a5dd-9003c003c60a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '202224f89f274d398b34f9653bd769cc'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:11.086945 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-39861c29-bb5b-4783-a5dd-9003c003c60a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '202224f89f274d398b34f9653bd769cc'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:11.098950 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:11.109919 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11dfb64b-323c-48bc-9fd9-6288ceff93cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:11.114325 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11dfb64b-323c-48bc-9fd9-6288ceff93cc admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:11.116164 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11dfb64b-323c-48bc-9fd9-6288ceff93cc admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:11.127245 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11dfb64b-323c-48bc-9fd9-6288ceff93cc admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:11.131444 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11dfb64b-323c-48bc-9fd9-6288ceff93cc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:11.143834 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 870/1767] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:47:10 2022] DELETE /identity/v3/users/202224f89f274d398b34f9653bd769cc => generated 0 bytes in 500 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:11.144023 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11dfb64b-323c-48bc-9fd9-6288ceff93cc admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:11.153500 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e3e1e4-e0e4-4ac4-a7a7-aea9f3f8b51d None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:11.154407 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e3e1e4-e0e4-4ac4-a7a7-aea9f3f8b51d None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:11.154882 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e3e1e4-e0e4-4ac4-a7a7-aea9f3f8b51d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:11.189172 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-19e3e1e4-e0e4-4ac4-a7a7-aea9f3f8b51d None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:11.191004 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-19e3e1e4-e0e4-4ac4-a7a7-aea9f3f8b51d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:11.209501 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 898/1768] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:47:10 2022] POST /identity/v3/users => generated 300 bytes in 347 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:11.214035 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11dfb64b-323c-48bc-9fd9-6288ceff93cc admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:11.220032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-11dfb64b-323c-48bc-9fd9-6288ceff93cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:11.239892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11dfb64b-323c-48bc-9fd9-6288ceff93cc admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:11.362639 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 871/1769] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:11 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:11.401763 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53518a2d-1abf-4803-9a2c-b58e13c03db1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:11.402583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53518a2d-1abf-4803-9a2c-b58e13c03db1 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:11.403309 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53518a2d-1abf-4803-9a2c-b58e13c03db1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:11.404393 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53518a2d-1abf-4803-9a2c-b58e13c03db1 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:11.407457 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53518a2d-1abf-4803-9a2c-b58e13c03db1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:11.411111 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53518a2d-1abf-4803-9a2c-b58e13c03db1 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:11.414360 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e3e1e4-e0e4-4ac4-a7a7-aea9f3f8b51d None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:11.416606 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-19e3e1e4-e0e4-4ac4-a7a7-aea9f3f8b51d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:11.446840 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 899/1770] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:11 2022] GET /identity/v3/roles => generated 1427 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:11.449841 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53518a2d-1abf-4803-9a2c-b58e13c03db1 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:11.474594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53518a2d-1abf-4803-9a2c-b58e13c03db1 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:11.484899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596be13a-9ffb-444d-97ff-b816d051a864 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:11.485738 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596be13a-9ffb-444d-97ff-b816d051a864 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:11.486028 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596be13a-9ffb-444d-97ff-b816d051a864 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:11.486308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596be13a-9ffb-444d-97ff-b816d051a864 admin admin] PATH_INFO: `/v3/projects/a5c0b8feed00420aa3913c5f9fe09089/users/24be85d36da34172a2b151d0adf40f73/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:11.571831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-596be13a-9ffb-444d-97ff-b816d051a864 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a5c0b8feed00420aa3913c5f9fe09089, user_id=24be85d36da34172a2b151d0adf40f73, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:11.585154 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-596be13a-9ffb-444d-97ff-b816d051a864 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:11.626896 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 900/1771] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:47:11 2022] PUT /identity/v3/projects/a5c0b8feed00420aa3913c5f9fe09089/users/24be85d36da34172a2b151d0adf40f73/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:11.628112 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596be13a-9ffb-444d-97ff-b816d051a864 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:11.636448 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e3e1e4-e0e4-4ac4-a7a7-aea9f3f8b51d None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:11.726338 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:11.760574 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596be13a-9ffb-444d-97ff-b816d051a864 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:11.767993 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b7d0ec-fe15-480c-8838-459254cf96f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:11.768780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b7d0ec-fe15-480c-8838-459254cf96f6 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:11.770541 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b7d0ec-fe15-480c-8838-459254cf96f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:11.770953 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b7d0ec-fe15-480c-8838-459254cf96f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:11.778053 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b7d0ec-fe15-480c-8838-459254cf96f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:11.780049 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b7d0ec-fe15-480c-8838-459254cf96f6 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:11.800032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 901/1772] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:11 2022] GET /identity/v3/roles => generated 1427 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:11.801041 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b7d0ec-fe15-480c-8838-459254cf96f6 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:11.850809 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b7d0ec-fe15-480c-8838-459254cf96f6 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:11.865255 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b25feac-fecc-4512-afe0-c0292604f0a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:11.866473 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b25feac-fecc-4512-afe0-c0292604f0a6 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:11.866903 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b25feac-fecc-4512-afe0-c0292604f0a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:11.867398 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b25feac-fecc-4512-afe0-c0292604f0a6 admin admin] PATH_INFO: `/v3/projects/a5c0b8feed00420aa3913c5f9fe09089/users/24be85d36da34172a2b151d0adf40f73/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:11.873967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b25feac-fecc-4512-afe0-c0292604f0a6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a5c0b8feed00420aa3913c5f9fe09089, user_id=24be85d36da34172a2b151d0adf40f73, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:11.880099 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b25feac-fecc-4512-afe0-c0292604f0a6 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:11.916321 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 902/1773] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:47:11 2022] PUT /identity/v3/projects/a5c0b8feed00420aa3913c5f9fe09089/users/24be85d36da34172a2b151d0adf40f73/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:11.921092 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a74934-bab7-4bc7-934c-20f7b5f60da9 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:11.921645 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a74934-bab7-4bc7-934c-20f7b5f60da9 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:11.922073 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a74934-bab7-4bc7-934c-20f7b5f60da9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:11.923538 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c367ac1a-7b44-4ab5-a9af-92b444bbd243 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:11.924203 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c367ac1a-7b44-4ab5-a9af-92b444bbd243 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:11.924485 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c367ac1a-7b44-4ab5-a9af-92b444bbd243 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:11.924761 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c367ac1a-7b44-4ab5-a9af-92b444bbd243 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:11.966467 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c367ac1a-7b44-4ab5-a9af-92b444bbd243 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:11.980060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c367ac1a-7b44-4ab5-a9af-92b444bbd243 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:12.047919 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 872/1774] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:11 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 634 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 15:47:12.065591 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-a9a74934-bab7-4bc7-934c-20f7b5f60da9 None None] MFA Rules not processed for user `24be85d36da34172a2b151d0adf40f73`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:12.067411 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-a9a74934-bab7-4bc7-934c-20f7b5f60da9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:12.261041 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 903/1775] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:11 2022] POST /identity/v3/auth/tokens => generated 3339 bytes in 344 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:12.284956 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c367ac1a-7b44-4ab5-a9af-92b444bbd243 service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:12.286770 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-c367ac1a-7b44-4ab5-a9af-92b444bbd243 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:12.512062 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c367ac1a-7b44-4ab5-a9af-92b444bbd243 service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:12.527809 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:DELETE=1,keystone:SELECT=99 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:12.744131 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e832ad-8f58-492f-a253-ff4123c7449c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:12.745615 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e832ad-8f58-492f-a253-ff4123c7449c service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:12.746066 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e832ad-8f58-492f-a253-ff4123c7449c service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:12.746526 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e832ad-8f58-492f-a253-ff4123c7449c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:12.806161 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e832ad-8f58-492f-a253-ff4123c7449c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:12.811584 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e832ad-8f58-492f-a253-ff4123c7449c service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:12.868390 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 873/1776] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:12 2022] GET /identity/v3/auth/tokens => generated 3339 bytes in 584 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:12.883893 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3bcd442-1959-4b44-809c-555451120723 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:12.883893 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3bcd442-1959-4b44-809c-555451120723 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:12.884657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3bcd442-1959-4b44-809c-555451120723 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:12.918064 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-a3bcd442-1959-4b44-809c-555451120723 None None] MFA Rules not processed for user `c058b07c1e9e443ebacbf38b1f13e58b`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:12.919987 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-a3bcd442-1959-4b44-809c-555451120723 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:13.103922 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e832ad-8f58-492f-a253-ff4123c7449c service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:13.275666 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 904/1777] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:12 2022] POST /identity/v3/auth/tokens => generated 3274 bytes in 392 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:13.397425 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e832ad-8f58-492f-a253-ff4123c7449c service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:13.416951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30089fdf-d4a5-4b69-8b3a-c5a0ab032986 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:13.417628 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30089fdf-d4a5-4b69-8b3a-c5a0ab032986 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:13.417923 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30089fdf-d4a5-4b69-8b3a-c5a0ab032986 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:13.418192 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30089fdf-d4a5-4b69-8b3a-c5a0ab032986 admin admin] PATH_INFO: `/v3/users/fc61eb142e1a49558fa736f8406a3c7b` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:13.424216 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30089fdf-d4a5-4b69-8b3a-c5a0ab032986 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fc61eb142e1a49558fa736f8406a3c7b)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:13.438837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30089fdf-d4a5-4b69-8b3a-c5a0ab032986 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:13.534995 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-30089fdf-d4a5-4b69-8b3a-c5a0ab032986 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fc61eb142e1a49558fa736f8406a3c7b'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:13.535903 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-30089fdf-d4a5-4b69-8b3a-c5a0ab032986 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fc61eb142e1a49558fa736f8406a3c7b'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:13.554110 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-30089fdf-d4a5-4b69-8b3a-c5a0ab032986 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fc61eb142e1a49558fa736f8406a3c7b'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:13.607232 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-30089fdf-d4a5-4b69-8b3a-c5a0ab032986 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fc61eb142e1a49558fa736f8406a3c7b'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:13.608099 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-30089fdf-d4a5-4b69-8b3a-c5a0ab032986 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fc61eb142e1a49558fa736f8406a3c7b'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:13.637864 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 874/1778] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:47:13 2022] DELETE /identity/v3/users/fc61eb142e1a49558fa736f8406a3c7b => generated 0 bytes in 535 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 15:47:13.642100 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393deda1-627f-4751-aace-35866947bc18 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:13.642396 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393deda1-627f-4751-aace-35866947bc18 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:13.642714 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393deda1-627f-4751-aace-35866947bc18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:13.699009 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-393deda1-627f-4751-aace-35866947bc18 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:13.700720 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-393deda1-627f-4751-aace-35866947bc18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:13.846395 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:13.901003 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 905/1779] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:13 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 257 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:47:13.942712 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9346c891-f0b9-4238-998a-dd1aeefb303f None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:13.943261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9346c891-f0b9-4238-998a-dd1aeefb303f None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:13.943261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9346c891-f0b9-4238-998a-dd1aeefb303f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:14.004293 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-9346c891-f0b9-4238-998a-dd1aeefb303f None None] MFA Rules not processed for user `38f7d31cf0f541f58af7d3d850af6472`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:14.005784 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-9346c891-f0b9-4238-998a-dd1aeefb303f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:14.011216 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393deda1-627f-4751-aace-35866947bc18 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:14.013485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-393deda1-627f-4751-aace-35866947bc18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:14.103989 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:14.219691 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393deda1-627f-4751-aace-35866947bc18 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:14.245402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768d7e42-dd19-47d0-b8f0-90ae3fa8b4f8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:14.247352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 875/1780] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:13 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 306 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:14.255462 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768d7e42-dd19-47d0-b8f0-90ae3fa8b4f8 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:14.255754 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768d7e42-dd19-47d0-b8f0-90ae3fa8b4f8 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:14.256032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768d7e42-dd19-47d0-b8f0-90ae3fa8b4f8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:14.304239 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9346c891-f0b9-4238-998a-dd1aeefb303f None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:14.307098 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-9346c891-f0b9-4238-998a-dd1aeefb303f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:14.319501 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768d7e42-dd19-47d0-b8f0-90ae3fa8b4f8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:14.321740 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768d7e42-dd19-47d0-b8f0-90ae3fa8b4f8 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:14.339453 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9346c891-f0b9-4238-998a-dd1aeefb303f None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:14.361257 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a704c1f-4fb0-44b0-b96f-0e2773955bb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:14.361913 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a704c1f-4fb0-44b0-b96f-0e2773955bb4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:14.362189 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a704c1f-4fb0-44b0-b96f-0e2773955bb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:14.362459 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a704c1f-4fb0-44b0-b96f-0e2773955bb4 admin admin] PATH_INFO: `/v3/users/eb6e8939a43c4b52b81d13351a108adb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:14.365519 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 906/1781] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:14 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 15:47:14.366287 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a704c1f-4fb0-44b0-b96f-0e2773955bb4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=eb6e8939a43c4b52b81d13351a108adb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:14.373147 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a704c1f-4fb0-44b0-b96f-0e2773955bb4 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:14.390878 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc35f98-9053-47f8-999e-93b3c122d14b None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:14.391161 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc35f98-9053-47f8-999e-93b3c122d14b None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:14.391440 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc35f98-9053-47f8-999e-93b3c122d14b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:14.448385 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-3cc35f98-9053-47f8-999e-93b3c122d14b None None] MFA Rules not processed for user `48939674c1fd4cf599691d510fcd8858`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:14.451388 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-3cc35f98-9053-47f8-999e-93b3c122d14b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:14.464251 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 907/1782] 10.222.0.62 () {58 vars in 1019 bytes} [Wed Aug 24 15:47:14 2022] POST /identity/v3/auth/tokens => generated 3131 bytes in 70 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:14.511204 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-4a704c1f-4fb0-44b0-b96f-0e2773955bb4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eb6e8939a43c4b52b81d13351a108adb'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:14.511875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-4a704c1f-4fb0-44b0-b96f-0e2773955bb4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eb6e8939a43c4b52b81d13351a108adb'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:14.519824 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-4a704c1f-4fb0-44b0-b96f-0e2773955bb4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eb6e8939a43c4b52b81d13351a108adb'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:14.577576 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc35f98-9053-47f8-999e-93b3c122d14b None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:14.579153 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-3cc35f98-9053-47f8-999e-93b3c122d14b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:14.594211 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-4a704c1f-4fb0-44b0-b96f-0e2773955bb4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eb6e8939a43c4b52b81d13351a108adb'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:14.595821 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-4a704c1f-4fb0-44b0-b96f-0e2773955bb4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eb6e8939a43c4b52b81d13351a108adb'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:14.613440 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc35f98-9053-47f8-999e-93b3c122d14b None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:14.626892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f612b0-8eab-43a8-a444-838eb58ae55c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:14.629327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f612b0-8eab-43a8-a444-838eb58ae55c service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:14.629903 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f612b0-8eab-43a8-a444-838eb58ae55c service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:14.630470 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f612b0-8eab-43a8-a444-838eb58ae55c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:14.645799 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-86f612b0-8eab-43a8-a444-838eb58ae55c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:14.655020 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 876/1783] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:47:14 2022] DELETE /identity/v3/users/eb6e8939a43c4b52b81d13351a108adb => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 15:47:14.661705 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc1dd13-772c-423f-b2c5-cf3e0f18312b None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:14.663157 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc1dd13-772c-423f-b2c5-cf3e0f18312b None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:14.663597 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc1dd13-772c-423f-b2c5-cf3e0f18312b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:14.691997 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86f612b0-8eab-43a8-a444-838eb58ae55c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:14.695730 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86f612b0-8eab-43a8-a444-838eb58ae55c service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:14.753858 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 908/1784] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:14 2022] GET /identity/v3/auth/tokens => generated 3131 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 15:47:14.755316 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-5fc1dd13-772c-423f-b2c5-cf3e0f18312b None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:14.759395 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-5fc1dd13-772c-423f-b2c5-cf3e0f18312b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:14.884582 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1988ae0-88f6-4e4c-ae2d-8129d42b04d3 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:14.885073 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1988ae0-88f6-4e4c-ae2d-8129d42b04d3 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:14.885747 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1988ae0-88f6-4e4c-ae2d-8129d42b04d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:14.975987 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-a1988ae0-88f6-4e4c-ae2d-8129d42b04d3 None None] MFA Rules not processed for user `7999201deb0d4f0391df21fdd0cdb60b`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:14.983424 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-a1988ae0-88f6-4e4c-ae2d-8129d42b04d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:15.010855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 877/1785] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:14 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 352 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:15.169281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc1dd13-772c-423f-b2c5-cf3e0f18312b None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:15.184027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-5fc1dd13-772c-423f-b2c5-cf3e0f18312b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:15.215505 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc1dd13-772c-423f-b2c5-cf3e0f18312b None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:15.219083 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 909/1786] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:14 2022] POST /identity/v3/auth/tokens => generated 3290 bytes in 337 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:15.222317 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1988ae0-88f6-4e4c-ae2d-8129d42b04d3 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:15.228225 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-a1988ae0-88f6-4e4c-ae2d-8129d42b04d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:15.231491 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afb0d7e-f9e0-4ea0-850c-8ae9d6543803 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:15.232458 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afb0d7e-f9e0-4ea0-850c-8ae9d6543803 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:15.232787 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afb0d7e-f9e0-4ea0-850c-8ae9d6543803 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:15.233139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afb0d7e-f9e0-4ea0-850c-8ae9d6543803 admin admin] PATH_INFO: `/v3/projects/ce7dce2a40e945f6be112ac8733b8b5f` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:15.237140 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0afb0d7e-f9e0-4ea0-850c-8ae9d6543803 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ce7dce2a40e945f6be112ac8733b8b5f)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:15.239695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0afb0d7e-f9e0-4ea0-850c-8ae9d6543803 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:15.347900 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:15.355903 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-0afb0d7e-f9e0-4ea0-850c-8ae9d6543803 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ce7dce2a40e945f6be112ac8733b8b5f', 'request_id': 'req-0afb0d7e-f9e0-4ea0-850c-8ae9d6543803'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:15.370748 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-0afb0d7e-f9e0-4ea0-850c-8ae9d6543803 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ce7dce2a40e945f6be112ac8733b8b5f', 'request_id': 'req-0afb0d7e-f9e0-4ea0-850c-8ae9d6543803'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:15.380932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-0afb0d7e-f9e0-4ea0-850c-8ae9d6543803 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ce7dce2a40e945f6be112ac8733b8b5f', 'request_id': 'req-0afb0d7e-f9e0-4ea0-850c-8ae9d6543803'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:15.435395 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-0afb0d7e-f9e0-4ea0-850c-8ae9d6543803 admin admin] The token cache is being invalidate because project ce7dce2a40e945f6be112ac8733b8b5f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:47:15.445577 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-0afb0d7e-f9e0-4ea0-850c-8ae9d6543803 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:15.448554 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-0afb0d7e-f9e0-4ea0-850c-8ae9d6543803 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:15.448837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1988ae0-88f6-4e4c-ae2d-8129d42b04d3 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:15.475041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 878/1787] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:47:15 2022] DELETE /identity/v3/projects/ce7dce2a40e945f6be112ac8733b8b5f => generated 0 bytes in 300 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 15:47:15.482346 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afb0d7e-f9e0-4ea0-850c-8ae9d6543803 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:15.482653 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:15.497258 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-0afb0d7e-f9e0-4ea0-850c-8ae9d6543803 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:15.582771 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31aa05e4-09c7-4584-9dba-62b4846c02c5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:15.583718 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31aa05e4-09c7-4584-9dba-62b4846c02c5 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:15.584104 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31aa05e4-09c7-4584-9dba-62b4846c02c5 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:15.584482 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31aa05e4-09c7-4584-9dba-62b4846c02c5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:15.588693 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-31aa05e4-09c7-4584-9dba-62b4846c02c5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:15.707378 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31aa05e4-09c7-4584-9dba-62b4846c02c5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:15.710132 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31aa05e4-09c7-4584-9dba-62b4846c02c5 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:15.724117 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afb0d7e-f9e0-4ea0-850c-8ae9d6543803 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:15.729429 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 910/1788] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:15 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 509 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:15.733753 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c8a902-9ad9-4a7d-ba7a-f942886dddf8 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:15.734600 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c8a902-9ad9-4a7d-ba7a-f942886dddf8 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:15.736157 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c8a902-9ad9-4a7d-ba7a-f942886dddf8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:15.755132 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f8ae06-c63a-45c7-97cf-4a281e67ea08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:15.755814 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f8ae06-c63a-45c7-97cf-4a281e67ea08 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:15.756110 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f8ae06-c63a-45c7-97cf-4a281e67ea08 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:15.757033 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f8ae06-c63a-45c7-97cf-4a281e67ea08 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:15.757854 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f8ae06-c63a-45c7-97cf-4a281e67ea08 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:15.761069 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f8ae06-c63a-45c7-97cf-4a281e67ea08 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:15.778423 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-e0c8a902-9ad9-4a7d-ba7a-f942886dddf8 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:15.781047 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-e0c8a902-9ad9-4a7d-ba7a-f942886dddf8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:15.799620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 911/1789] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:15 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 69 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:15.801463 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a112de9d-89f7-47d1-8da0-a891d363d102 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:15.801863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a112de9d-89f7-47d1-8da0-a891d363d102 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:15.802243 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a112de9d-89f7-47d1-8da0-a891d363d102 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:15.829267 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-a112de9d-89f7-47d1-8da0-a891d363d102 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:15.834721 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-a112de9d-89f7-47d1-8da0-a891d363d102 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:15.850233 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 912/1790] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:15 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:15.856201 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a112de9d-89f7-47d1-8da0-a891d363d102 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:15.868345 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a112de9d-89f7-47d1-8da0-a891d363d102 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:15.873666 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62514631-0c4c-4ad7-b13b-61b6411fceed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:15.875795 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62514631-0c4c-4ad7-b13b-61b6411fceed admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:15.877027 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62514631-0c4c-4ad7-b13b-61b6411fceed admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:15.877529 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62514631-0c4c-4ad7-b13b-61b6411fceed admin admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:15.879478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62514631-0c4c-4ad7-b13b-61b6411fceed admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:47:15.880821 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62514631-0c4c-4ad7-b13b-61b6411fceed admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:15.881457 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 879/1791] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:47:15 2022] POST /identity/v3/projects => generated 404 bytes in 406 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:15.883924 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62514631-0c4c-4ad7-b13b-61b6411fceed admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:15.891357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f8ae06-c63a-45c7-97cf-4a281e67ea08 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:15.914312 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 913/1792] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:47:15 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:47:15.916322 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62514631-0c4c-4ad7-b13b-61b6411fceed admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:15.930636 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62514631-0c4c-4ad7-b13b-61b6411fceed admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:16.029568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb22473a-c6e1-496c-9fe4-d6304f7d0d41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:16.030381 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb22473a-c6e1-496c-9fe4-d6304f7d0d41 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:16.030879 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb22473a-c6e1-496c-9fe4-d6304f7d0d41 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:16.031313 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb22473a-c6e1-496c-9fe4-d6304f7d0d41 admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:16.032301 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb22473a-c6e1-496c-9fe4-d6304f7d0d41 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:16.034728 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb22473a-c6e1-496c-9fe4-d6304f7d0d41 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:16.038410 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f8ae06-c63a-45c7-97cf-4a281e67ea08 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:16.050275 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cb8e7a-2afc-4ced-a9fd-8b1202369e92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:16.050810 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cb8e7a-2afc-4ced-a9fd-8b1202369e92 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:16.051119 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cb8e7a-2afc-4ced-a9fd-8b1202369e92 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:16.051364 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cb8e7a-2afc-4ced-a9fd-8b1202369e92 admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:16.052039 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9cb8e7a-2afc-4ced-a9fd-8b1202369e92 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:16.054552 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9cb8e7a-2afc-4ced-a9fd-8b1202369e92 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:16.136677 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 914/1793] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:47:15 2022] POST /identity/v3/projects => generated 364 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:16.137708 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb22473a-c6e1-496c-9fe4-d6304f7d0d41 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:16.142357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 880/1794] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:47:15 2022] POST /identity/v3/users => generated 309 bytes in 251 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:16.146464 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cb8e7a-2afc-4ced-a9fd-8b1202369e92 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:16.169237 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cb8e7a-2afc-4ced-a9fd-8b1202369e92 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:16.193382 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:16.237480 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb22473a-c6e1-496c-9fe4-d6304f7d0d41 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:16.243770 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd97e4c-8f4a-459f-a273-890d0dae6623 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:16.244550 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd97e4c-8f4a-459f-a273-890d0dae6623 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:16.244897 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd97e4c-8f4a-459f-a273-890d0dae6623 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:16.245561 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd97e4c-8f4a-459f-a273-890d0dae6623 admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:16.246346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd97e4c-8f4a-459f-a273-890d0dae6623 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:16.248740 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd97e4c-8f4a-459f-a273-890d0dae6623 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:16.290534 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=91,keystone:DELETE=4,keystone:INSERT=5 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:16.292680 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 915/1795] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:47:16 2022] POST /identity/v3/users => generated 283 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:16.297490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd97e4c-8f4a-459f-a273-890d0dae6623 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:16.322460 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4c679f-d28c-44e3-82ad-7b0149daf412 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:16.323094 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4c679f-d28c-44e3-82ad-7b0149daf412 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:16.323376 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4c679f-d28c-44e3-82ad-7b0149daf412 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:16.323673 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4c679f-d28c-44e3-82ad-7b0149daf412 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:16.324342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4c679f-d28c-44e3-82ad-7b0149daf412 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:16.325968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4c679f-d28c-44e3-82ad-7b0149daf412 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:16.330327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd97e4c-8f4a-459f-a273-890d0dae6623 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:16.345519 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a633275c-532f-4274-a453-c6652e6907fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:16.346196 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a633275c-532f-4274-a453-c6652e6907fa admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:16.346530 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a633275c-532f-4274-a453-c6652e6907fa admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:16.346829 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a633275c-532f-4274-a453-c6652e6907fa admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:16.347472 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a633275c-532f-4274-a453-c6652e6907fa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:16.349116 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a633275c-532f-4274-a453-c6652e6907fa admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:16.352757 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 881/1796] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:16 2022] GET /identity/v3/roles => generated 1427 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:16.357087 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4c679f-d28c-44e3-82ad-7b0149daf412 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:16.375886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4c679f-d28c-44e3-82ad-7b0149daf412 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:16.385909 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 916/1797] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:16 2022] GET /identity/v3/roles => generated 1427 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:16.389940 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a633275c-532f-4274-a453-c6652e6907fa admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:16.393622 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304adaa4-d261-49f2-96bd-725ff675fc3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:16.394197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304adaa4-d261-49f2-96bd-725ff675fc3e admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:16.394497 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304adaa4-d261-49f2-96bd-725ff675fc3e admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:16.394772 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304adaa4-d261-49f2-96bd-725ff675fc3e admin admin] PATH_INFO: `/v3/projects/188a0715b8094f30a3de10f4b410d397/users/c1adb6fdaabe4e2296dfcd5ccc117662/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:16.419084 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a633275c-532f-4274-a453-c6652e6907fa admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:16.441096 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8caf11-3d17-40e1-ade1-eb03e74b0afe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:16.441976 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8caf11-3d17-40e1-ade1-eb03e74b0afe admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:16.442258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8caf11-3d17-40e1-ade1-eb03e74b0afe admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:16.442537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8caf11-3d17-40e1-ade1-eb03e74b0afe admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:16.443186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8caf11-3d17-40e1-ade1-eb03e74b0afe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:16.446264 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-304adaa4-d261-49f2-96bd-725ff675fc3e admin admin] RBAC: Authorizing `identity:create_grant(project_id=188a0715b8094f30a3de10f4b410d397, user_id=c1adb6fdaabe4e2296dfcd5ccc117662, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:16.450229 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8caf11-3d17-40e1-ade1-eb03e74b0afe admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:16.456880 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-304adaa4-d261-49f2-96bd-725ff675fc3e admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:16.487728 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 882/1798] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:47:16 2022] PUT /identity/v3/projects/188a0715b8094f30a3de10f4b410d397/users/c1adb6fdaabe4e2296dfcd5ccc117662/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:16.491666 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304adaa4-d261-49f2-96bd-725ff675fc3e admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:16.517389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 917/1799] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:16 2022] GET /identity/v3/roles => generated 1427 bytes in 128 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:16.518476 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8caf11-3d17-40e1-ade1-eb03e74b0afe admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:16.543174 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8caf11-3d17-40e1-ade1-eb03e74b0afe admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:16.631541 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304adaa4-d261-49f2-96bd-725ff675fc3e admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:16.641243 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2abd95a-1967-4a88-b073-16b81e816712 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:16.642830 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2abd95a-1967-4a88-b073-16b81e816712 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:16.643357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2abd95a-1967-4a88-b073-16b81e816712 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:16.643895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2abd95a-1967-4a88-b073-16b81e816712 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:16.646456 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2abd95a-1967-4a88-b073-16b81e816712 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:16.648823 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2abd95a-1967-4a88-b073-16b81e816712 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:16.716497 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 883/1800] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:16 2022] GET /identity/v3/roles => generated 1427 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:16.718657 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2abd95a-1967-4a88-b073-16b81e816712 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:16.743153 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a9993e-fefa-469b-9b5e-9ca237da0a7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:16.749618 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a9993e-fefa-469b-9b5e-9ca237da0a7e admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:16.749755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a9993e-fefa-469b-9b5e-9ca237da0a7e admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:16.750040 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a9993e-fefa-469b-9b5e-9ca237da0a7e admin admin] PATH_INFO: `/v3/projects/574b1f5308274f1c812e9752f2d2e7fc/users/8554e5739a994c10a6edecfd37765bc5/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:16.772791 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2abd95a-1967-4a88-b073-16b81e816712 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:16.797529 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1335d10-5840-4a15-b99b-acf9ffc7f375 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:16.799155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1335d10-5840-4a15-b99b-acf9ffc7f375 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:16.799455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1335d10-5840-4a15-b99b-acf9ffc7f375 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:16.800431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1335d10-5840-4a15-b99b-acf9ffc7f375 admin admin] PATH_INFO: `/v3/projects/188a0715b8094f30a3de10f4b410d397/users/c1adb6fdaabe4e2296dfcd5ccc117662/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:16.806076 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1335d10-5840-4a15-b99b-acf9ffc7f375 admin admin] RBAC: Authorizing `identity:create_grant(project_id=188a0715b8094f30a3de10f4b410d397, user_id=c1adb6fdaabe4e2296dfcd5ccc117662, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:16.815958 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1335d10-5840-4a15-b99b-acf9ffc7f375 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:16.816422 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a9993e-fefa-469b-9b5e-9ca237da0a7e admin admin] RBAC: Authorizing `identity:create_grant(project_id=574b1f5308274f1c812e9752f2d2e7fc, user_id=8554e5739a994c10a6edecfd37765bc5, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:16.819560 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a9993e-fefa-469b-9b5e-9ca237da0a7e admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:16.856560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 884/1801] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:47:16 2022] PUT /identity/v3/projects/188a0715b8094f30a3de10f4b410d397/users/c1adb6fdaabe4e2296dfcd5ccc117662/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:16.870579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 918/1802] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:47:16 2022] PUT /identity/v3/projects/574b1f5308274f1c812e9752f2d2e7fc/users/8554e5739a994c10a6edecfd37765bc5/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 353 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:16.881026 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239f37d0-90a2-4fde-bf16-f9e4c48113f1 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:16.881481 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239f37d0-90a2-4fde-bf16-f9e4c48113f1 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:16.882012 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239f37d0-90a2-4fde-bf16-f9e4c48113f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:16.971374 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-239f37d0-90a2-4fde-bf16-f9e4c48113f1 None None] MFA Rules not processed for user `8554e5739a994c10a6edecfd37765bc5`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:16.975704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-239f37d0-90a2-4fde-bf16-f9e4c48113f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:17.235558 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a9993e-fefa-469b-9b5e-9ca237da0a7e admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:17.236792 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 885/1803] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:16 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 356 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:47:17.455428 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a9993e-fefa-469b-9b5e-9ca237da0a7e admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:17.462004 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412ba191-ac93-4b72-b82e-8d9c6f34b903 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:17.462698 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412ba191-ac93-4b72-b82e-8d9c6f34b903 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:17.462981 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412ba191-ac93-4b72-b82e-8d9c6f34b903 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:17.463305 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412ba191-ac93-4b72-b82e-8d9c6f34b903 admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:17.464985 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412ba191-ac93-4b72-b82e-8d9c6f34b903 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:17.467143 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412ba191-ac93-4b72-b82e-8d9c6f34b903 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:17.589418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 919/1804] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:47:17 2022] POST /identity/v3/projects => generated 368 bytes in 355 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 15:47:17.590510 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239f37d0-90a2-4fde-bf16-f9e4c48113f1 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:17.626199 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239f37d0-90a2-4fde-bf16-f9e4c48113f1 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:17.832826 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:17.972504 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-324df8f7-4c7f-4b65-82a6-589bc21f736d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:17.973312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324df8f7-4c7f-4b65-82a6-589bc21f736d admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:17.973697 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324df8f7-4c7f-4b65-82a6-589bc21f736d admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:17.975798 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324df8f7-4c7f-4b65-82a6-589bc21f736d admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:17.976817 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-324df8f7-4c7f-4b65-82a6-589bc21f736d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:17.983217 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-324df8f7-4c7f-4b65-82a6-589bc21f736d admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:18.062330 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412ba191-ac93-4b72-b82e-8d9c6f34b903 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:18.071462 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 886/1805] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:47:17 2022] POST /identity/v3/users => generated 291 bytes in 481 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:18.072151 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:18.078519 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412ba191-ac93-4b72-b82e-8d9c6f34b903 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:18.105733 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66792a2e-7d7e-4dda-be37-80d4c88d11e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:18.106819 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66792a2e-7d7e-4dda-be37-80d4c88d11e2 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:18.107495 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66792a2e-7d7e-4dda-be37-80d4c88d11e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:18.108960 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66792a2e-7d7e-4dda-be37-80d4c88d11e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:18.109935 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66792a2e-7d7e-4dda-be37-80d4c88d11e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:18.113366 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66792a2e-7d7e-4dda-be37-80d4c88d11e2 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:18.160611 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-324df8f7-4c7f-4b65-82a6-589bc21f736d admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:18.160882 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 920/1806] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:18 2022] GET /identity/v3/roles => generated 1427 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 15:47:18.174451 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-324df8f7-4c7f-4b65-82a6-589bc21f736d admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:18.213244 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9daf5c3-01ad-469e-9568-bd0e02ca3c3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:18.214013 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9daf5c3-01ad-469e-9568-bd0e02ca3c3e admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:18.214240 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9daf5c3-01ad-469e-9568-bd0e02ca3c3e admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:18.214524 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9daf5c3-01ad-469e-9568-bd0e02ca3c3e admin admin] PATH_INFO: `/v3/projects/f2ed6eaddf0f445ab4437da183776596/users/a130b5f8a0274c3084d2f406bbffede6/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:18.311250 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9daf5c3-01ad-469e-9568-bd0e02ca3c3e admin admin] RBAC: Authorizing `identity:create_grant(project_id=f2ed6eaddf0f445ab4437da183776596, user_id=a130b5f8a0274c3084d2f406bbffede6, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:18.314114 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9daf5c3-01ad-469e-9568-bd0e02ca3c3e admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:18.371805 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 887/1807] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:47:18 2022] PUT /identity/v3/projects/f2ed6eaddf0f445ab4437da183776596/users/a130b5f8a0274c3084d2f406bbffede6/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:18.373024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66792a2e-7d7e-4dda-be37-80d4c88d11e2 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:18.388672 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66792a2e-7d7e-4dda-be37-80d4c88d11e2 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:18.522892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd955c7f-abe1-46c0-9863-21d03a27c6b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:18.539836 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd955c7f-abe1-46c0-9863-21d03a27c6b0 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:18.544544 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd955c7f-abe1-46c0-9863-21d03a27c6b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:18.545131 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd955c7f-abe1-46c0-9863-21d03a27c6b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:18.546045 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd955c7f-abe1-46c0-9863-21d03a27c6b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:18.547969 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd955c7f-abe1-46c0-9863-21d03a27c6b0 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:18.575326 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 921/1808] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:18 2022] GET /identity/v3/roles => generated 1427 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:18.579114 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9daf5c3-01ad-469e-9568-bd0e02ca3c3e admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:18.607103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9daf5c3-01ad-469e-9568-bd0e02ca3c3e admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:18.634863 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db3c399-5973-4cea-9c36-f73cccfe8c1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:18.635554 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db3c399-5973-4cea-9c36-f73cccfe8c1e admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:18.636043 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db3c399-5973-4cea-9c36-f73cccfe8c1e admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:18.636328 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db3c399-5973-4cea-9c36-f73cccfe8c1e admin admin] PATH_INFO: `/v3/projects/f2ed6eaddf0f445ab4437da183776596/users/a130b5f8a0274c3084d2f406bbffede6/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:18.641989 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db3c399-5973-4cea-9c36-f73cccfe8c1e admin admin] RBAC: Authorizing `identity:create_grant(project_id=f2ed6eaddf0f445ab4437da183776596, user_id=a130b5f8a0274c3084d2f406bbffede6, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:18.651305 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db3c399-5973-4cea-9c36-f73cccfe8c1e admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:18.687868 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 888/1809] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:47:18 2022] PUT /identity/v3/projects/f2ed6eaddf0f445ab4437da183776596/users/a130b5f8a0274c3084d2f406bbffede6/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:18.699396 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ba197b-ede5-49cf-9263-50a7321b72f1 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:18.699869 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ba197b-ede5-49cf-9263-50a7321b72f1 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:18.700310 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ba197b-ede5-49cf-9263-50a7321b72f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:18.759103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db3c399-5973-4cea-9c36-f73cccfe8c1e admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:18.762756 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-5db3c399-5973-4cea-9c36-f73cccfe8c1e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:18.814616 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-55ba197b-ede5-49cf-9263-50a7321b72f1 None None] MFA Rules not processed for user `a130b5f8a0274c3084d2f406bbffede6`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:18.819798 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-55ba197b-ede5-49cf-9263-50a7321b72f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:18.989081 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db3c399-5973-4cea-9c36-f73cccfe8c1e admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:19.010295 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c54eb7-14a0-461f-b683-42622f3f98c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:19.010949 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c54eb7-14a0-461f-b683-42622f3f98c2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:19.011231 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c54eb7-14a0-461f-b683-42622f3f98c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:19.011504 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c54eb7-14a0-461f-b683-42622f3f98c2 admin admin] PATH_INFO: `/v3/projects/a71f2a925d0a4dc085ba518e5779b93d` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:19.013381 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c54eb7-14a0-461f-b683-42622f3f98c2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a71f2a925d0a4dc085ba518e5779b93d)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:19.015273 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c54eb7-14a0-461f-b683-42622f3f98c2 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:19.137989 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 922/1810] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:18 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 440 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:19.150818 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ba197b-ede5-49cf-9263-50a7321b72f1 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:19.152630 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-55ba197b-ede5-49cf-9263-50a7321b72f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:19.157665 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-60c54eb7-14a0-461f-b683-42622f3f98c2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a71f2a925d0a4dc085ba518e5779b93d', 'request_id': 'req-60c54eb7-14a0-461f-b683-42622f3f98c2'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:19.176284 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-60c54eb7-14a0-461f-b683-42622f3f98c2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a71f2a925d0a4dc085ba518e5779b93d', 'request_id': 'req-60c54eb7-14a0-461f-b683-42622f3f98c2'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:19.196330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-60c54eb7-14a0-461f-b683-42622f3f98c2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a71f2a925d0a4dc085ba518e5779b93d', 'request_id': 'req-60c54eb7-14a0-461f-b683-42622f3f98c2'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:19.221966 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-60c54eb7-14a0-461f-b683-42622f3f98c2 admin admin] The token cache is being invalidate because project a71f2a925d0a4dc085ba518e5779b93d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:47:19.222164 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-60c54eb7-14a0-461f-b683-42622f3f98c2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:19.222912 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-60c54eb7-14a0-461f-b683-42622f3f98c2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:19.224681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 889/1811] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:47:18 2022] DELETE /identity/v3/projects/a71f2a925d0a4dc085ba518e5779b93d => generated 0 bytes in 466 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:19.230122 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c54eb7-14a0-461f-b683-42622f3f98c2 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:19.231598 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-60c54eb7-14a0-461f-b683-42622f3f98c2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:19.268355 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=87,keystone:INSERT=6,keystone:DELETE=7 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:19.299296 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ba197b-ede5-49cf-9263-50a7321b72f1 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:19.380583 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c54eb7-14a0-461f-b683-42622f3f98c2 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:19.381666 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:19.392592 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f84ad56-255d-4f62-ab8f-723c2e91d5d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:19.393724 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f84ad56-255d-4f62-ab8f-723c2e91d5d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:19.394814 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f84ad56-255d-4f62-ab8f-723c2e91d5d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:19.395312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f84ad56-255d-4f62-ab8f-723c2e91d5d5 admin admin] PATH_INFO: `/v3/users/24be85d36da34172a2b151d0adf40f73` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:19.397169 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f84ad56-255d-4f62-ab8f-723c2e91d5d5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=24be85d36da34172a2b151d0adf40f73)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:19.401712 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f84ad56-255d-4f62-ab8f-723c2e91d5d5 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:19.438755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cded0d1b-0ea7-4ff5-bd66-bcaa88ab003e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:19.439557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cded0d1b-0ea7-4ff5-bd66-bcaa88ab003e service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:19.440298 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cded0d1b-0ea7-4ff5-bd66-bcaa88ab003e service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:19.440298 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cded0d1b-0ea7-4ff5-bd66-bcaa88ab003e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:19.442972 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-cded0d1b-0ea7-4ff5-bd66-bcaa88ab003e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:19.537495 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-4f84ad56-255d-4f62-ab8f-723c2e91d5d5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '24be85d36da34172a2b151d0adf40f73'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:19.538240 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-4f84ad56-255d-4f62-ab8f-723c2e91d5d5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '24be85d36da34172a2b151d0adf40f73'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:19.567535 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-4f84ad56-255d-4f62-ab8f-723c2e91d5d5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '24be85d36da34172a2b151d0adf40f73'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:19.595112 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-4f84ad56-255d-4f62-ab8f-723c2e91d5d5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '24be85d36da34172a2b151d0adf40f73'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:19.596347 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-4f84ad56-255d-4f62-ab8f-723c2e91d5d5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '24be85d36da34172a2b151d0adf40f73'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:19.630774 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 890/1812] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:47:19 2022] DELETE /identity/v3/users/24be85d36da34172a2b151d0adf40f73 => generated 0 bytes in 403 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:19.652991 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cded0d1b-0ea7-4ff5-bd66-bcaa88ab003e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:19.656411 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cded0d1b-0ea7-4ff5-bd66-bcaa88ab003e service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:19.682088 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 923/1813] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:19 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 532 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:19.981317 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5959e0-f3d0-427c-8b73-a3f0c99ab2aa None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:19.981689 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5959e0-f3d0-427c-8b73-a3f0c99ab2aa None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:19.982040 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5959e0-f3d0-427c-8b73-a3f0c99ab2aa None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:19.983930 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 891/1814] 10.222.0.102 () {54 vars in 926 bytes} [Wed Aug 24 15:47:19 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:47:20.128669 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cded0d1b-0ea7-4ff5-bd66-bcaa88ab003e service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:20.130593 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-cded0d1b-0ea7-4ff5-bd66-bcaa88ab003e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:20.152409 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c78b373-811f-4e93-92ac-409bf8a5fccb None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:20.152824 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c78b373-811f-4e93-92ac-409bf8a5fccb None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:20.153802 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c78b373-811f-4e93-92ac-409bf8a5fccb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:20.184361 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cded0d1b-0ea7-4ff5-bd66-bcaa88ab003e service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:20.231927 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-5c78b373-811f-4e93-92ac-409bf8a5fccb None None] MFA Rules not processed for user `e0d544379fe44438ad10289b45124374`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:20.233313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-5c78b373-811f-4e93-92ac-409bf8a5fccb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:20.345975 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8821541-0f20-4fa9-8e28-e2a94c14b173 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:20.346883 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8821541-0f20-4fa9-8e28-e2a94c14b173 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:20.347510 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8821541-0f20-4fa9-8e28-e2a94c14b173 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:20.348036 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8821541-0f20-4fa9-8e28-e2a94c14b173 admin admin] PATH_INFO: `/v3/projects/bf1040f6c7a3412dbf0cb067e736c2cc` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:20.349693 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8821541-0f20-4fa9-8e28-e2a94c14b173 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bf1040f6c7a3412dbf0cb067e736c2cc)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:20.353573 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8821541-0f20-4fa9-8e28-e2a94c14b173 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:20.474783 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 892/1815] 10.222.0.102 () {58 vars in 1023 bytes} [Wed Aug 24 15:47:20 2022] POST /identity/v3/auth/tokens => generated 3382 bytes in 323 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:20.479844 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08325e4e-5cfa-4d55-b2af-2b0e8e668fa1 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:20.480228 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08325e4e-5cfa-4d55-b2af-2b0e8e668fa1 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:20.480719 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08325e4e-5cfa-4d55-b2af-2b0e8e668fa1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:20.527662 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-f8821541-0f20-4fa9-8e28-e2a94c14b173 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bf1040f6c7a3412dbf0cb067e736c2cc', 'request_id': 'req-f8821541-0f20-4fa9-8e28-e2a94c14b173'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:20.542604 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-f8821541-0f20-4fa9-8e28-e2a94c14b173 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bf1040f6c7a3412dbf0cb067e736c2cc', 'request_id': 'req-f8821541-0f20-4fa9-8e28-e2a94c14b173'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:20.560403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-f8821541-0f20-4fa9-8e28-e2a94c14b173 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bf1040f6c7a3412dbf0cb067e736c2cc', 'request_id': 'req-f8821541-0f20-4fa9-8e28-e2a94c14b173'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:20.582138 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-f8821541-0f20-4fa9-8e28-e2a94c14b173 admin admin] The token cache is being invalidate because project bf1040f6c7a3412dbf0cb067e736c2cc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:47:20.582438 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-f8821541-0f20-4fa9-8e28-e2a94c14b173 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:20.585639 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-f8821541-0f20-4fa9-8e28-e2a94c14b173 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:20.588550 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 924/1816] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:47:20 2022] DELETE /identity/v3/projects/bf1040f6c7a3412dbf0cb067e736c2cc => generated 0 bytes in 460 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:20.593248 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8821541-0f20-4fa9-8e28-e2a94c14b173 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:20.597284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-f8821541-0f20-4fa9-8e28-e2a94c14b173 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:20.628940 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:20.634920 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-08325e4e-5cfa-4d55-b2af-2b0e8e668fa1 None None] MFA Rules not processed for user `c1adb6fdaabe4e2296dfcd5ccc117662`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:20.641501 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-08325e4e-5cfa-4d55-b2af-2b0e8e668fa1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:20.696829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:20.795006 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 893/1817] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:20 2022] POST /identity/v3/auth/tokens => generated 3357 bytes in 315 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:20.795006 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08325e4e-5cfa-4d55-b2af-2b0e8e668fa1 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:20.795936 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-08325e4e-5cfa-4d55-b2af-2b0e8e668fa1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:20.863102 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8821541-0f20-4fa9-8e28-e2a94c14b173 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:20.888427 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4466347c-d53e-41ee-80d3-dbb5b2478fec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:20.891150 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4466347c-d53e-41ee-80d3-dbb5b2478fec admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:20.891483 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4466347c-d53e-41ee-80d3-dbb5b2478fec admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:20.891808 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4466347c-d53e-41ee-80d3-dbb5b2478fec admin admin] PATH_INFO: `/v3/users/cef7ea25a97248df9ae51cbd9980f1b7` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:20.900266 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4466347c-d53e-41ee-80d3-dbb5b2478fec admin admin] RBAC: Authorizing `identity:delete_user(user_id=cef7ea25a97248df9ae51cbd9980f1b7)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:20.904832 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4466347c-d53e-41ee-80d3-dbb5b2478fec admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:20.974394 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08325e4e-5cfa-4d55-b2af-2b0e8e668fa1 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:21.011616 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee837e2-9f0e-4b1d-b580-05a22e0dc4b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:21.013680 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee837e2-9f0e-4b1d-b580-05a22e0dc4b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:21.021478 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee837e2-9f0e-4b1d-b580-05a22e0dc4b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:21.022828 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee837e2-9f0e-4b1d-b580-05a22e0dc4b1 admin admin] PATH_INFO: `/v3/projects/a5c0b8feed00420aa3913c5f9fe09089` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:21.026217 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee837e2-9f0e-4b1d-b580-05a22e0dc4b1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a5c0b8feed00420aa3913c5f9fe09089)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:21.028534 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee837e2-9f0e-4b1d-b580-05a22e0dc4b1 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:21.051289 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4466347c-d53e-41ee-80d3-dbb5b2478fec admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cef7ea25a97248df9ae51cbd9980f1b7'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:21.052286 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4466347c-d53e-41ee-80d3-dbb5b2478fec admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cef7ea25a97248df9ae51cbd9980f1b7'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:21.054134 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4466347c-d53e-41ee-80d3-dbb5b2478fec admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cef7ea25a97248df9ae51cbd9980f1b7'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:21.070023 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4466347c-d53e-41ee-80d3-dbb5b2478fec admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cef7ea25a97248df9ae51cbd9980f1b7'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:21.117003 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4466347c-d53e-41ee-80d3-dbb5b2478fec admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cef7ea25a97248df9ae51cbd9980f1b7'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:21.183099 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 925/1818] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:47:20 2022] DELETE /identity/v3/users/cef7ea25a97248df9ae51cbd9980f1b7 => generated 0 bytes in 590 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 15:47:21.189434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-aee837e2-9f0e-4b1d-b580-05a22e0dc4b1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a5c0b8feed00420aa3913c5f9fe09089', 'request_id': 'req-aee837e2-9f0e-4b1d-b580-05a22e0dc4b1'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:21.189753 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4466347c-d53e-41ee-80d3-dbb5b2478fec admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:21.192591 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-4466347c-d53e-41ee-80d3-dbb5b2478fec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:21.216680 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-aee837e2-9f0e-4b1d-b580-05a22e0dc4b1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a5c0b8feed00420aa3913c5f9fe09089', 'request_id': 'req-aee837e2-9f0e-4b1d-b580-05a22e0dc4b1'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:21.236264 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-aee837e2-9f0e-4b1d-b580-05a22e0dc4b1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a5c0b8feed00420aa3913c5f9fe09089', 'request_id': 'req-aee837e2-9f0e-4b1d-b580-05a22e0dc4b1'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:21.287199 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-aee837e2-9f0e-4b1d-b580-05a22e0dc4b1 admin admin] The token cache is being invalidate because project a5c0b8feed00420aa3913c5f9fe09089 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:47:21.287510 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-aee837e2-9f0e-4b1d-b580-05a22e0dc4b1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:21.290269 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-aee837e2-9f0e-4b1d-b580-05a22e0dc4b1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:21.294314 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 894/1819] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:47:20 2022] DELETE /identity/v3/projects/a5c0b8feed00420aa3913c5f9fe09089 => generated 0 bytes in 503 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:21.298874 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e1bdb5-9a57-49f3-9eb2-0b8b13091e29 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:21.299207 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e1bdb5-9a57-49f3-9eb2-0b8b13091e29 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:21.299489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e1bdb5-9a57-49f3-9eb2-0b8b13091e29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:21.344838 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-15e1bdb5-9a57-49f3-9eb2-0b8b13091e29 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:21.346087 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-15e1bdb5-9a57-49f3-9eb2-0b8b13091e29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:21.480451 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:21.481988 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4466347c-d53e-41ee-80d3-dbb5b2478fec admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:21.512542 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116eb184-5842-4613-8592-519a8d85ce39 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:21.513212 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116eb184-5842-4613-8592-519a8d85ce39 service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:21.513705 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116eb184-5842-4613-8592-519a8d85ce39 service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:21.513995 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116eb184-5842-4613-8592-519a8d85ce39 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:21.517521 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-116eb184-5842-4613-8592-519a8d85ce39 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:21.562956 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-116eb184-5842-4613-8592-519a8d85ce39 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:21.564607 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-116eb184-5842-4613-8592-519a8d85ce39 service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:21.589035 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 895/1820] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:21 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 291 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:21.598843 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bfd635-ab6f-4c76-8f9d-b0beaeedff30 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:21.602220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bfd635-ab6f-4c76-8f9d-b0beaeedff30 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:21.602220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bfd635-ab6f-4c76-8f9d-b0beaeedff30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:21.605124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 926/1821] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:21 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 421 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:21.760168 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-f8bfd635-ab6f-4c76-8f9d-b0beaeedff30 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:21.762780 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-f8bfd635-ab6f-4c76-8f9d-b0beaeedff30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:21.818234 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 896/1822] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:21 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:47:21.818362 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116eb184-5842-4613-8592-519a8d85ce39 service nova] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:21.837806 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116eb184-5842-4613-8592-519a8d85ce39 service nova] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:21.857552 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc84fdf-b119-4c50-ace7-6b0e722ceea5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:21.858187 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc84fdf-b119-4c50-ace7-6b0e722ceea5 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:21.858482 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc84fdf-b119-4c50-ace7-6b0e722ceea5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:21.858830 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc84fdf-b119-4c50-ace7-6b0e722ceea5 admin admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:21.859523 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc84fdf-b119-4c50-ace7-6b0e722ceea5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:47:21.859897 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc84fdf-b119-4c50-ace7-6b0e722ceea5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:21.861937 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc84fdf-b119-4c50-ace7-6b0e722ceea5 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:21.900935 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 927/1823] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:47:21 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:47:21.907049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8bfd635-ab6f-4c76-8f9d-b0beaeedff30 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:21.927315 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8bfd635-ab6f-4c76-8f9d-b0beaeedff30 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:21.956814 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9ea5b4-e865-46cc-8963-a41f31f2f8b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:21.962196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9ea5b4-e865-46cc-8963-a41f31f2f8b0 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:21.962520 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9ea5b4-e865-46cc-8963-a41f31f2f8b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:21.968724 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9ea5b4-e865-46cc-8963-a41f31f2f8b0 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:21.970858 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9ea5b4-e865-46cc-8963-a41f31f2f8b0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:21.975022 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9ea5b4-e865-46cc-8963-a41f31f2f8b0 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:22.043783 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 897/1824] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:47:21 2022] POST /identity/v3/projects => generated 390 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:22.048062 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc84fdf-b119-4c50-ace7-6b0e722ceea5 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:22.056132 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:22.065881 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc84fdf-b119-4c50-ace7-6b0e722ceea5 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:22.202633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e03ad6-853e-4f72-b625-4f7e2803a52c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:22.203209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e03ad6-853e-4f72-b625-4f7e2803a52c admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:22.203728 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e03ad6-853e-4f72-b625-4f7e2803a52c admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:22.204052 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e03ad6-853e-4f72-b625-4f7e2803a52c admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:22.204771 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e03ad6-853e-4f72-b625-4f7e2803a52c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:22.206945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e03ad6-853e-4f72-b625-4f7e2803a52c admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:22.269226 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 928/1825] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:47:22 2022] POST /identity/v3/users => generated 296 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:22.271171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9ea5b4-e865-46cc-8963-a41f31f2f8b0 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:22.311238 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9ea5b4-e865-46cc-8963-a41f31f2f8b0 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:22.332782 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d0e323-251e-497e-a3a1-ffa355577770 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:22.337318 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d0e323-251e-497e-a3a1-ffa355577770 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:22.337948 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d0e323-251e-497e-a3a1-ffa355577770 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:22.338487 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d0e323-251e-497e-a3a1-ffa355577770 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:22.342841 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d0e323-251e-497e-a3a1-ffa355577770 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:22.348258 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d0e323-251e-497e-a3a1-ffa355577770 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:22.386379 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e03ad6-853e-4f72-b625-4f7e2803a52c admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:22.387207 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 898/1826] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:22 2022] GET /identity/v3/roles => generated 1427 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:22.410524 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e03ad6-853e-4f72-b625-4f7e2803a52c admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:22.421880 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77beb7ab-c502-44c2-b2ff-8e885bd90103 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:22.422521 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77beb7ab-c502-44c2-b2ff-8e885bd90103 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:22.422835 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77beb7ab-c502-44c2-b2ff-8e885bd90103 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:22.423133 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77beb7ab-c502-44c2-b2ff-8e885bd90103 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:22.424436 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77beb7ab-c502-44c2-b2ff-8e885bd90103 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:22.425504 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77beb7ab-c502-44c2-b2ff-8e885bd90103 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:22.458069 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 929/1827] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:22 2022] GET /identity/v3/roles => generated 1427 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 15:47:22.464467 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d0e323-251e-497e-a3a1-ffa355577770 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:22.482670 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d0e323-251e-497e-a3a1-ffa355577770 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:22.494116 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d33873-90f2-4377-be39-fadfbdf0b1fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:22.494878 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d33873-90f2-4377-be39-fadfbdf0b1fb admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:22.495151 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d33873-90f2-4377-be39-fadfbdf0b1fb admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:22.495447 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d33873-90f2-4377-be39-fadfbdf0b1fb admin admin] PATH_INFO: `/v3/projects/ae84f1465e1444a5afde3ad7883cef33/users/c88b89aca2de410baab5024eee90996a/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:22.528544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d33873-90f2-4377-be39-fadfbdf0b1fb admin admin] RBAC: Authorizing `identity:create_grant(project_id=ae84f1465e1444a5afde3ad7883cef33, user_id=c88b89aca2de410baab5024eee90996a, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:22.542953 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d33873-90f2-4377-be39-fadfbdf0b1fb admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:22.574742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 899/1828] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:47:22 2022] PUT /identity/v3/projects/ae84f1465e1444a5afde3ad7883cef33/users/c88b89aca2de410baab5024eee90996a/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:22.590151 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4ca94a-d0f2-48b9-bf5d-64db8d2580f7 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:22.590769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4ca94a-d0f2-48b9-bf5d-64db8d2580f7 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:22.591224 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4ca94a-d0f2-48b9-bf5d-64db8d2580f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:22.649611 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-ed4ca94a-d0f2-48b9-bf5d-64db8d2580f7 None None] MFA Rules not processed for user `c88b89aca2de410baab5024eee90996a`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:22.653239 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-ed4ca94a-d0f2-48b9-bf5d-64db8d2580f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:22.698232 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d33873-90f2-4377-be39-fadfbdf0b1fb admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:22.701410 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-86d33873-90f2-4377-be39-fadfbdf0b1fb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:22.863975 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 930/1829] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:22 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 275 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:22.870924 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed4ca94a-d0f2-48b9-bf5d-64db8d2580f7 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:22.914692 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d33873-90f2-4377-be39-fadfbdf0b1fb admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:22.931382 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3459c2c-5191-4843-9c92-ff27b26d01f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:22.932108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3459c2c-5191-4843-9c92-ff27b26d01f7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:22.932391 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3459c2c-5191-4843-9c92-ff27b26d01f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:22.932687 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3459c2c-5191-4843-9c92-ff27b26d01f7 admin admin] PATH_INFO: `/v3/projects/f0f68ee2560e449b8d77c3eeff196e53` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:22.934097 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3459c2c-5191-4843-9c92-ff27b26d01f7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f0f68ee2560e449b8d77c3eeff196e53)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:22.936773 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3459c2c-5191-4843-9c92-ff27b26d01f7 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:23.081236 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d3459c2c-5191-4843-9c92-ff27b26d01f7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f0f68ee2560e449b8d77c3eeff196e53', 'request_id': 'req-d3459c2c-5191-4843-9c92-ff27b26d01f7'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:23.114142 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed4ca94a-d0f2-48b9-bf5d-64db8d2580f7 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:23.123179 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d3459c2c-5191-4843-9c92-ff27b26d01f7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f0f68ee2560e449b8d77c3eeff196e53', 'request_id': 'req-d3459c2c-5191-4843-9c92-ff27b26d01f7'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:23.132108 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:23.159004 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d3459c2c-5191-4843-9c92-ff27b26d01f7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f0f68ee2560e449b8d77c3eeff196e53', 'request_id': 'req-d3459c2c-5191-4843-9c92-ff27b26d01f7'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:23.188886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d3459c2c-5191-4843-9c92-ff27b26d01f7 admin admin] The token cache is being invalidate because project f0f68ee2560e449b8d77c3eeff196e53 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:47:23.189195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d3459c2c-5191-4843-9c92-ff27b26d01f7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:23.189949 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d3459c2c-5191-4843-9c92-ff27b26d01f7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:23.192183 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 900/1830] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:47:22 2022] DELETE /identity/v3/projects/f0f68ee2560e449b8d77c3eeff196e53 => generated 0 bytes in 495 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:23.193798 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3459c2c-5191-4843-9c92-ff27b26d01f7 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:23.196986 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-d3459c2c-5191-4843-9c92-ff27b26d01f7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:23.209571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:23.223089 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c84922b-5803-410e-9bc4-a13fd453d60c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:23.223964 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c84922b-5803-410e-9bc4-a13fd453d60c admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:23.224111 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c84922b-5803-410e-9bc4-a13fd453d60c admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:23.224370 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c84922b-5803-410e-9bc4-a13fd453d60c admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:23.227106 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c84922b-5803-410e-9bc4-a13fd453d60c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:23.231412 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c84922b-5803-410e-9bc4-a13fd453d60c admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:23.279932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 931/1831] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:47:22 2022] POST /identity/v3/projects => generated 388 bytes in 412 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:23.281151 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c84922b-5803-410e-9bc4-a13fd453d60c admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:23.283259 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-5c84922b-5803-410e-9bc4-a13fd453d60c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:23.302034 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3459c2c-5191-4843-9c92-ff27b26d01f7 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:23.427260 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c84922b-5803-410e-9bc4-a13fd453d60c admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:23.435058 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8e0ed0-568f-4f66-93d0-5299a2966827 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:23.435988 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8e0ed0-568f-4f66-93d0-5299a2966827 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:23.436427 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8e0ed0-568f-4f66-93d0-5299a2966827 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:23.436868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8e0ed0-568f-4f66-93d0-5299a2966827 admin admin] PATH_INFO: `/v3/users/5e54c3c38a5444dcb8b377410e88b101` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:23.443151 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8e0ed0-568f-4f66-93d0-5299a2966827 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5e54c3c38a5444dcb8b377410e88b101)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:23.445922 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8e0ed0-568f-4f66-93d0-5299a2966827 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:23.454626 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0534615-7c21-49fa-94eb-5166e3e93e22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:23.455285 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0534615-7c21-49fa-94eb-5166e3e93e22 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:23.455773 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0534615-7c21-49fa-94eb-5166e3e93e22 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:23.456156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0534615-7c21-49fa-94eb-5166e3e93e22 admin admin] PATH_INFO: `/v3/users/8554e5739a994c10a6edecfd37765bc5` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:23.457661 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0534615-7c21-49fa-94eb-5166e3e93e22 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8554e5739a994c10a6edecfd37765bc5)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:23.459797 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0534615-7c21-49fa-94eb-5166e3e93e22 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:23.550597 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-a0534615-7c21-49fa-94eb-5166e3e93e22 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8554e5739a994c10a6edecfd37765bc5'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:23.551372 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-a0534615-7c21-49fa-94eb-5166e3e93e22 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8554e5739a994c10a6edecfd37765bc5'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:23.569280 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-3a8e0ed0-568f-4f66-93d0-5299a2966827 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5e54c3c38a5444dcb8b377410e88b101'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:23.576566 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-a0534615-7c21-49fa-94eb-5166e3e93e22 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8554e5739a994c10a6edecfd37765bc5'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:23.580855 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-3a8e0ed0-568f-4f66-93d0-5299a2966827 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5e54c3c38a5444dcb8b377410e88b101'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:23.581587 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-3a8e0ed0-568f-4f66-93d0-5299a2966827 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5e54c3c38a5444dcb8b377410e88b101'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:23.593798 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-3a8e0ed0-568f-4f66-93d0-5299a2966827 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5e54c3c38a5444dcb8b377410e88b101'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:23.605638 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-a0534615-7c21-49fa-94eb-5166e3e93e22 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8554e5739a994c10a6edecfd37765bc5'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:23.606487 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-a0534615-7c21-49fa-94eb-5166e3e93e22 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8554e5739a994c10a6edecfd37765bc5'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:23.614488 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-3a8e0ed0-568f-4f66-93d0-5299a2966827 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5e54c3c38a5444dcb8b377410e88b101'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:23.629854 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 901/1832] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:47:23 2022] DELETE /identity/v3/users/8554e5739a994c10a6edecfd37765bc5 => generated 0 bytes in 434 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:23.630964 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0534615-7c21-49fa-94eb-5166e3e93e22 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:23.634194 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-a0534615-7c21-49fa-94eb-5166e3e93e22 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:23.647067 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 932/1833] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:47:23 2022] DELETE /identity/v3/users/5e54c3c38a5444dcb8b377410e88b101 => generated 0 bytes in 366 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:23.648353 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abc0812-dc31-44a6-b634-05816fe99f69 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:23.649159 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abc0812-dc31-44a6-b634-05816fe99f69 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:23.649430 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abc0812-dc31-44a6-b634-05816fe99f69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:23.676558 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-8abc0812-dc31-44a6-b634-05816fe99f69 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:23.677808 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-8abc0812-dc31-44a6-b634-05816fe99f69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:23.748962 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:23.765771 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:23.857378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0534615-7c21-49fa-94eb-5166e3e93e22 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:23.862473 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 933/1834] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:23 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:23.871338 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82aa7c63-2502-4b17-b826-d43ea61197c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:23.872260 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82aa7c63-2502-4b17-b826-d43ea61197c9 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:23.872619 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82aa7c63-2502-4b17-b826-d43ea61197c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:23.872892 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82aa7c63-2502-4b17-b826-d43ea61197c9 admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:23.873672 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82aa7c63-2502-4b17-b826-d43ea61197c9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:23.882265 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82aa7c63-2502-4b17-b826-d43ea61197c9 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:23.918939 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 902/1835] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:47:23 2022] POST /identity/v3/users => generated 301 bytes in 288 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:23.921241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8abc0812-dc31-44a6-b634-05816fe99f69 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:23.934169 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8abc0812-dc31-44a6-b634-05816fe99f69 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:23.938400 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0acbf4-0caa-4079-8631-fcdb303998bd None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:23.939269 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0acbf4-0caa-4079-8631-fcdb303998bd None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:23.939795 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0acbf4-0caa-4079-8631-fcdb303998bd None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:23.941621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 903/1836] 10.222.0.102 () {54 vars in 926 bytes} [Wed Aug 24 15:47:23 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:47:23.947806 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06cc901-1cd2-4b39-a2c2-334d35edcfe1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:23.948782 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06cc901-1cd2-4b39-a2c2-334d35edcfe1 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:23.949533 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06cc901-1cd2-4b39-a2c2-334d35edcfe1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:23.950086 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06cc901-1cd2-4b39-a2c2-334d35edcfe1 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:23.952101 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06cc901-1cd2-4b39-a2c2-334d35edcfe1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:23.954046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06cc901-1cd2-4b39-a2c2-334d35edcfe1 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:23.960392 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57521c1-8fd4-416b-8a55-33d8c9921ea1 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:23.961092 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57521c1-8fd4-416b-8a55-33d8c9921ea1 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:23.961643 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57521c1-8fd4-416b-8a55-33d8c9921ea1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:24.000462 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-e57521c1-8fd4-416b-8a55-33d8c9921ea1 None None] MFA Rules not processed for user `0ee0002a76c3471f819b37fef510ef99`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:24.003936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 934/1837] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:23 2022] GET /identity/v3/roles => generated 1427 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:24.003936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06cc901-1cd2-4b39-a2c2-334d35edcfe1 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:24.004400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-c06cc901-1cd2-4b39-a2c2-334d35edcfe1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:24.006658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-e57521c1-8fd4-416b-8a55-33d8c9921ea1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:24.201842 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 904/1838] 10.222.0.102 () {58 vars in 1023 bytes} [Wed Aug 24 15:47:23 2022] POST /identity/v3/auth/tokens => generated 3318 bytes in 243 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:24.203624 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57521c1-8fd4-416b-8a55-33d8c9921ea1 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:24.220910 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06cc901-1cd2-4b39-a2c2-334d35edcfe1 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:24.237620 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57521c1-8fd4-416b-8a55-33d8c9921ea1 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:24.239877 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4be31b6-656d-41b6-8a77-e951ac9f7973 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:24.240566 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4be31b6-656d-41b6-8a77-e951ac9f7973 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:24.240925 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4be31b6-656d-41b6-8a77-e951ac9f7973 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:24.241270 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4be31b6-656d-41b6-8a77-e951ac9f7973 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:24.244723 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca95b308-5192-4d5f-8a0a-3efba15e9c63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:24.245797 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca95b308-5192-4d5f-8a0a-3efba15e9c63 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:24.246270 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca95b308-5192-4d5f-8a0a-3efba15e9c63 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:24.246807 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca95b308-5192-4d5f-8a0a-3efba15e9c63 admin admin] PATH_INFO: `/v3/projects/3df5583daab94bcd8fdc3806b48645d1/users/31ccfb389bc8400e85421f68d3608145/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:24.260909 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4be31b6-656d-41b6-8a77-e951ac9f7973 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:24.263402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4be31b6-656d-41b6-8a77-e951ac9f7973 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:24.315231 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 935/1839] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:24 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:24.319514 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4be31b6-656d-41b6-8a77-e951ac9f7973 service neutron] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:24.322187 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-a4be31b6-656d-41b6-8a77-e951ac9f7973 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:24.324003 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca95b308-5192-4d5f-8a0a-3efba15e9c63 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3df5583daab94bcd8fdc3806b48645d1, user_id=31ccfb389bc8400e85421f68d3608145, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:24.335028 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca95b308-5192-4d5f-8a0a-3efba15e9c63 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:24.363084 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4be31b6-656d-41b6-8a77-e951ac9f7973 service neutron] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:24.368439 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 905/1840] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:47:24 2022] PUT /identity/v3/projects/3df5583daab94bcd8fdc3806b48645d1/users/31ccfb389bc8400e85421f68d3608145/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:24.376619 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca95b308-5192-4d5f-8a0a-3efba15e9c63 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:24.382460 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-ca95b308-5192-4d5f-8a0a-3efba15e9c63 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:24.562767 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818f906f-511e-4f11-a979-b14c03d50e6d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0ee0002a76c3471f819b37fef510ef99', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:24.563412 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818f906f-511e-4f11-a979-b14c03d50e6d service placement] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:24.563717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818f906f-511e-4f11-a979-b14c03d50e6d service placement] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:24.563991 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818f906f-511e-4f11-a979-b14c03d50e6d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:24.579260 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca95b308-5192-4d5f-8a0a-3efba15e9c63 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:24.587357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818f906f-511e-4f11-a979-b14c03d50e6d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:24.589236 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818f906f-511e-4f11-a979-b14c03d50e6d service placement] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:24.599761 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93856a8-dd21-470e-b2d6-88455c46dcb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:24.600650 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93856a8-dd21-470e-b2d6-88455c46dcb2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:24.601087 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93856a8-dd21-470e-b2d6-88455c46dcb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:24.601416 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93856a8-dd21-470e-b2d6-88455c46dcb2 admin admin] PATH_INFO: `/v3/projects/7c1409f1315c43dbb6c4080fc3ed5ca0` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:24.607022 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93856a8-dd21-470e-b2d6-88455c46dcb2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7c1409f1315c43dbb6c4080fc3ed5ca0)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:24.612371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93856a8-dd21-470e-b2d6-88455c46dcb2 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:24.625020 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 936/1841] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:24 2022] GET /identity/v3/auth/tokens => generated 3318 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:24.628962 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818f906f-511e-4f11-a979-b14c03d50e6d service placement] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:24.653416 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818f906f-511e-4f11-a979-b14c03d50e6d service placement] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:24.668072 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33f9662-72bf-46d7-971e-6abb840b86b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:24.668701 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33f9662-72bf-46d7-971e-6abb840b86b4 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:24.668978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33f9662-72bf-46d7-971e-6abb840b86b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:24.669250 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33f9662-72bf-46d7-971e-6abb840b86b4 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:24.669909 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c33f9662-72bf-46d7-971e-6abb840b86b4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:24.671569 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c33f9662-72bf-46d7-971e-6abb840b86b4 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:24.709310 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 937/1842] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:24 2022] GET /identity/v3/roles => generated 1427 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:24.711922 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33f9662-72bf-46d7-971e-6abb840b86b4 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:24.713929 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=93,keystone:INSERT=4,keystone:DELETE=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:24.738700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-e93856a8-dd21-470e-b2d6-88455c46dcb2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7c1409f1315c43dbb6c4080fc3ed5ca0', 'request_id': 'req-e93856a8-dd21-470e-b2d6-88455c46dcb2'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:24.743225 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33f9662-72bf-46d7-971e-6abb840b86b4 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:24.759825 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-e93856a8-dd21-470e-b2d6-88455c46dcb2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7c1409f1315c43dbb6c4080fc3ed5ca0', 'request_id': 'req-e93856a8-dd21-470e-b2d6-88455c46dcb2'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:24.777904 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-e93856a8-dd21-470e-b2d6-88455c46dcb2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7c1409f1315c43dbb6c4080fc3ed5ca0', 'request_id': 'req-e93856a8-dd21-470e-b2d6-88455c46dcb2'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:24.798434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-e93856a8-dd21-470e-b2d6-88455c46dcb2 admin admin] The token cache is being invalidate because project 7c1409f1315c43dbb6c4080fc3ed5ca0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:47:24.799048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-e93856a8-dd21-470e-b2d6-88455c46dcb2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:24.799980 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-e93856a8-dd21-470e-b2d6-88455c46dcb2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:24.807725 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 906/1843] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:47:24 2022] DELETE /identity/v3/projects/7c1409f1315c43dbb6c4080fc3ed5ca0 => generated 0 bytes in 433 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:24.809676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e93df20-3a76-49c9-b1e1-1ff9c0da1c6b None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:24.810121 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e93df20-3a76-49c9-b1e1-1ff9c0da1c6b None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:24.810516 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e93df20-3a76-49c9-b1e1-1ff9c0da1c6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:24.812451 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:24.849105 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-1e93df20-3a76-49c9-b1e1-1ff9c0da1c6b None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:24.852488 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-1e93df20-3a76-49c9-b1e1-1ff9c0da1c6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:24.852718 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a00761-e6bc-4b2b-a135-0f4c44ceb9bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:24.853379 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a00761-e6bc-4b2b-a135-0f4c44ceb9bc admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:24.854138 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a00761-e6bc-4b2b-a135-0f4c44ceb9bc admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:24.854557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a00761-e6bc-4b2b-a135-0f4c44ceb9bc admin admin] PATH_INFO: `/v3/projects/3df5583daab94bcd8fdc3806b48645d1/users/31ccfb389bc8400e85421f68d3608145/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:24.858271 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a00761-e6bc-4b2b-a135-0f4c44ceb9bc admin admin] RBAC: Authorizing `identity:create_grant(project_id=3df5583daab94bcd8fdc3806b48645d1, user_id=31ccfb389bc8400e85421f68d3608145, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:24.862536 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a00761-e6bc-4b2b-a135-0f4c44ceb9bc admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:24.871951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 907/1844] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:24 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:24.878277 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 938/1845] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:47:24 2022] PUT /identity/v3/projects/3df5583daab94bcd8fdc3806b48645d1/users/31ccfb389bc8400e85421f68d3608145/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:24.881274 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42de1bf5-a6b2-494c-a1a1-f591c3a02aac None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:24.881811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42de1bf5-a6b2-494c-a1a1-f591c3a02aac None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:24.882284 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42de1bf5-a6b2-494c-a1a1-f591c3a02aac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:24.890490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bf556f-0282-45fd-9e27-d70973e06c24 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:24.891042 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bf556f-0282-45fd-9e27-d70973e06c24 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:24.891042 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bf556f-0282-45fd-9e27-d70973e06c24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:24.923509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-42de1bf5-a6b2-494c-a1a1-f591c3a02aac None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:24.924934 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-42de1bf5-a6b2-494c-a1a1-f591c3a02aac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:24.931203 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-08bf556f-0282-45fd-9e27-d70973e06c24 None None] MFA Rules not processed for user `31ccfb389bc8400e85421f68d3608145`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:24.933265 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-08bf556f-0282-45fd-9e27-d70973e06c24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:25.018448 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 908/1846] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:24 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:25.022651 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42de1bf5-a6b2-494c-a1a1-f591c3a02aac None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:25.043803 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42de1bf5-a6b2-494c-a1a1-f591c3a02aac None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:25.059139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9737947f-c8f0-4c1d-9501-504514566e89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:25.061360 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9737947f-c8f0-4c1d-9501-504514566e89 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:25.061972 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9737947f-c8f0-4c1d-9501-504514566e89 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:25.062442 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9737947f-c8f0-4c1d-9501-504514566e89 admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:25.063367 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9737947f-c8f0-4c1d-9501-504514566e89 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:47:25.064465 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9737947f-c8f0-4c1d-9501-504514566e89 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:25.070875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9737947f-c8f0-4c1d-9501-504514566e89 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:25.075974 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 939/1847] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:24 2022] POST /identity/v3/auth/tokens => generated 3341 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:47:25.075974 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08bf556f-0282-45fd-9e27-d70973e06c24 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:25.076671 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-08bf556f-0282-45fd-9e27-d70973e06c24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:25.093405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08bf556f-0282-45fd-9e27-d70973e06c24 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:25.100324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d578d8-3807-469e-ae1c-2deaa6033f6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:25.100961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d578d8-3807-469e-ae1c-2deaa6033f6c admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:25.102083 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d578d8-3807-469e-ae1c-2deaa6033f6c admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:25.103712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d578d8-3807-469e-ae1c-2deaa6033f6c admin admin] PATH_INFO: `/v3/projects/574b1f5308274f1c812e9752f2d2e7fc` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:25.104163 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d578d8-3807-469e-ae1c-2deaa6033f6c admin admin] RBAC: Authorizing `identity:delete_project(project_id=574b1f5308274f1c812e9752f2d2e7fc)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:25.107300 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d578d8-3807-469e-ae1c-2deaa6033f6c admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:25.115789 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 909/1848] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:47:25 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:47:25.116793 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9737947f-c8f0-4c1d-9501-504514566e89 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:25.119697 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-9737947f-c8f0-4c1d-9501-504514566e89 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:25.172643 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-62d578d8-3807-469e-ae1c-2deaa6033f6c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '574b1f5308274f1c812e9752f2d2e7fc', 'request_id': 'req-62d578d8-3807-469e-ae1c-2deaa6033f6c'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:25.181931 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-62d578d8-3807-469e-ae1c-2deaa6033f6c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '574b1f5308274f1c812e9752f2d2e7fc', 'request_id': 'req-62d578d8-3807-469e-ae1c-2deaa6033f6c'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:25.208300 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-62d578d8-3807-469e-ae1c-2deaa6033f6c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '574b1f5308274f1c812e9752f2d2e7fc', 'request_id': 'req-62d578d8-3807-469e-ae1c-2deaa6033f6c'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:25.213731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9737947f-c8f0-4c1d-9501-504514566e89 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:25.234404 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-62d578d8-3807-469e-ae1c-2deaa6033f6c admin admin] The token cache is being invalidate because project 574b1f5308274f1c812e9752f2d2e7fc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:47:25.235006 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-62d578d8-3807-469e-ae1c-2deaa6033f6c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:25.236876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-62d578d8-3807-469e-ae1c-2deaa6033f6c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:25.240656 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 940/1849] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:47:25 2022] DELETE /identity/v3/projects/574b1f5308274f1c812e9752f2d2e7fc => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:25.246014 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d578d8-3807-469e-ae1c-2deaa6033f6c admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:25.249865 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-62d578d8-3807-469e-ae1c-2deaa6033f6c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:25.308063 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62cfb949-1d4b-4af8-a101-be21723dffd4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:25.308933 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cfb949-1d4b-4af8-a101-be21723dffd4 service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:25.309216 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cfb949-1d4b-4af8-a101-be21723dffd4 service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:25.310276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cfb949-1d4b-4af8-a101-be21723dffd4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:25.312873 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-62cfb949-1d4b-4af8-a101-be21723dffd4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:25.334904 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:25.336777 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:25.374297 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d578d8-3807-469e-ae1c-2deaa6033f6c admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:25.382440 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badb91fc-d91d-4063-baf0-846ccd698fa5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:25.382440 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badb91fc-d91d-4063-baf0-846ccd698fa5 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:25.383248 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badb91fc-d91d-4063-baf0-846ccd698fa5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:25.384515 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badb91fc-d91d-4063-baf0-846ccd698fa5 admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:25.385418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-badb91fc-d91d-4063-baf0-846ccd698fa5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:25.388960 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-badb91fc-d91d-4063-baf0-846ccd698fa5 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:25.439889 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62cfb949-1d4b-4af8-a101-be21723dffd4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:25.440079 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 941/1850] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:47:25 2022] POST /identity/v3/projects => generated 366 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:25.441025 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badb91fc-d91d-4063-baf0-846ccd698fa5 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:25.442527 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62cfb949-1d4b-4af8-a101-be21723dffd4 service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:25.443756 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-badb91fc-d91d-4063-baf0-846ccd698fa5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:25.482571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 910/1851] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:25 2022] GET /identity/v3/auth/tokens => generated 3341 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:25.485549 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62cfb949-1d4b-4af8-a101-be21723dffd4 service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:25.525907 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62cfb949-1d4b-4af8-a101-be21723dffd4 service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:25.657038 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badb91fc-d91d-4063-baf0-846ccd698fa5 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:25.672750 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210df9b2-b4ac-498b-a4a9-14dfd6fecf1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:25.673256 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210df9b2-b4ac-498b-a4a9-14dfd6fecf1b admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:25.673543 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210df9b2-b4ac-498b-a4a9-14dfd6fecf1b admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:25.678238 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210df9b2-b4ac-498b-a4a9-14dfd6fecf1b admin admin] PATH_INFO: `/v3/users/a130b5f8a0274c3084d2f406bbffede6` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:25.680081 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-210df9b2-b4ac-498b-a4a9-14dfd6fecf1b admin admin] RBAC: Authorizing `identity:delete_user(user_id=a130b5f8a0274c3084d2f406bbffede6)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:25.685048 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-210df9b2-b4ac-498b-a4a9-14dfd6fecf1b admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:25.729178 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e8df56-20d0-4ea8-9afa-91313db889ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:25.729818 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e8df56-20d0-4ea8-9afa-91313db889ad admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:25.733347 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e8df56-20d0-4ea8-9afa-91313db889ad admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:25.733747 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e8df56-20d0-4ea8-9afa-91313db889ad admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:25.734640 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e8df56-20d0-4ea8-9afa-91313db889ad admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:25.740197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e8df56-20d0-4ea8-9afa-91313db889ad admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:25.785147 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 911/1852] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:47:25 2022] POST /identity/v3/users => generated 284 bytes in 302 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:25.792232 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e8df56-20d0-4ea8-9afa-91313db889ad admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:25.796570 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-40e8df56-20d0-4ea8-9afa-91313db889ad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:25.802307 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-210df9b2-b4ac-498b-a4a9-14dfd6fecf1b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a130b5f8a0274c3084d2f406bbffede6'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:25.803700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-210df9b2-b4ac-498b-a4a9-14dfd6fecf1b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a130b5f8a0274c3084d2f406bbffede6'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:25.806348 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-210df9b2-b4ac-498b-a4a9-14dfd6fecf1b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a130b5f8a0274c3084d2f406bbffede6'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:25.822709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-210df9b2-b4ac-498b-a4a9-14dfd6fecf1b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a130b5f8a0274c3084d2f406bbffede6'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:25.833983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e8df56-20d0-4ea8-9afa-91313db889ad admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:25.847355 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f561efc7-465f-4de9-b69e-c6166acba885 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:25.847978 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f561efc7-465f-4de9-b69e-c6166acba885 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:25.848256 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f561efc7-465f-4de9-b69e-c6166acba885 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:25.848990 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-210df9b2-b4ac-498b-a4a9-14dfd6fecf1b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a130b5f8a0274c3084d2f406bbffede6'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:25.850456 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f561efc7-465f-4de9-b69e-c6166acba885 admin admin] PATH_INFO: `/v3/users/c88b89aca2de410baab5024eee90996a` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:25.853093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f561efc7-465f-4de9-b69e-c6166acba885 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c88b89aca2de410baab5024eee90996a)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:25.855657 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f561efc7-465f-4de9-b69e-c6166acba885 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:25.879389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 942/1853] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:47:25 2022] DELETE /identity/v3/users/a130b5f8a0274c3084d2f406bbffede6 => generated 0 bytes in 439 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:25.880474 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210df9b2-b4ac-498b-a4a9-14dfd6fecf1b admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:25.886394 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-210df9b2-b4ac-498b-a4a9-14dfd6fecf1b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:25.975595 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:25.986914 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-f561efc7-465f-4de9-b69e-c6166acba885 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c88b89aca2de410baab5024eee90996a'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:25.987993 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-f561efc7-465f-4de9-b69e-c6166acba885 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c88b89aca2de410baab5024eee90996a'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:26.005011 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-f561efc7-465f-4de9-b69e-c6166acba885 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c88b89aca2de410baab5024eee90996a'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:26.044099 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-f561efc7-465f-4de9-b69e-c6166acba885 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c88b89aca2de410baab5024eee90996a'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:26.044849 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-f561efc7-465f-4de9-b69e-c6166acba885 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c88b89aca2de410baab5024eee90996a'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:26.063773 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 912/1854] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:47:25 2022] DELETE /identity/v3/users/c88b89aca2de410baab5024eee90996a => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:26.067429 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210df9b2-b4ac-498b-a4a9-14dfd6fecf1b admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:26.068747 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f71852-8101-4126-be86-ec8da0a42146 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:26.069469 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f71852-8101-4126-be86-ec8da0a42146 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:26.071034 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f71852-8101-4126-be86-ec8da0a42146 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:26.120959 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-40f71852-8101-4126-be86-ec8da0a42146 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:26.125561 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-40f71852-8101-4126-be86-ec8da0a42146 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:26.152746 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:26.159835 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36d8fe3-46c0-4ea1-bf73-409570ebb093 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:26.160543 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36d8fe3-46c0-4ea1-bf73-409570ebb093 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:26.160928 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36d8fe3-46c0-4ea1-bf73-409570ebb093 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:26.161873 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36d8fe3-46c0-4ea1-bf73-409570ebb093 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:26.162101 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d36d8fe3-46c0-4ea1-bf73-409570ebb093 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:26.166540 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d36d8fe3-46c0-4ea1-bf73-409570ebb093 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:26.192585 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 943/1855] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:25 2022] GET /identity/v3/roles => generated 1427 bytes in 312 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:26.196964 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36d8fe3-46c0-4ea1-bf73-409570ebb093 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:26.199511 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-d36d8fe3-46c0-4ea1-bf73-409570ebb093 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:26.214029 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36d8fe3-46c0-4ea1-bf73-409570ebb093 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:26.219909 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0ef2e4-a838-436d-a4fc-0fbe7842a35b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:26.220531 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0ef2e4-a838-436d-a4fc-0fbe7842a35b admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:26.220890 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0ef2e4-a838-436d-a4fc-0fbe7842a35b admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:26.221238 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0ef2e4-a838-436d-a4fc-0fbe7842a35b admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:26.221964 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f0ef2e4-a838-436d-a4fc-0fbe7842a35b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:26.223741 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f0ef2e4-a838-436d-a4fc-0fbe7842a35b admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:26.236123 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 944/1856] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:26 2022] GET /identity/v3/roles => generated 1427 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:26.239094 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0ef2e4-a838-436d-a4fc-0fbe7842a35b admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:26.257372 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0ef2e4-a838-436d-a4fc-0fbe7842a35b admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:26.266928 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a855785f-0ee3-457d-9102-5f53f819cbf1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:26.267099 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 913/1857] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:26 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:26.267191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a855785f-0ee3-457d-9102-5f53f819cbf1 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:26.267191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a855785f-0ee3-457d-9102-5f53f819cbf1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:26.267191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a855785f-0ee3-457d-9102-5f53f819cbf1 admin admin] PATH_INFO: `/v3/projects/e57aad5c0f1147a3a95508e38d10cf78/users/a39c02c655db4db08180a82f2082798b/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:26.273560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f71852-8101-4126-be86-ec8da0a42146 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:26.274969 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-40f71852-8101-4126-be86-ec8da0a42146 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:26.286613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a855785f-0ee3-457d-9102-5f53f819cbf1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e57aad5c0f1147a3a95508e38d10cf78, user_id=a39c02c655db4db08180a82f2082798b, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:26.290798 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a855785f-0ee3-457d-9102-5f53f819cbf1 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:26.306660 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 945/1858] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:47:26 2022] PUT /identity/v3/projects/e57aad5c0f1147a3a95508e38d10cf78/users/a39c02c655db4db08180a82f2082798b/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 15:47:26.318470 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680c1600-86e1-4ade-be21-47911511ed20 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:26.318770 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680c1600-86e1-4ade-be21-47911511ed20 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:26.319034 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680c1600-86e1-4ade-be21-47911511ed20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:26.349428 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-680c1600-86e1-4ade-be21-47911511ed20 None None] MFA Rules not processed for user `a39c02c655db4db08180a82f2082798b`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:26.350675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-680c1600-86e1-4ade-be21-47911511ed20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:26.358990 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f71852-8101-4126-be86-ec8da0a42146 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:26.420539 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e27f87-bc58-4707-96d4-d653f96ac613 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:26.421522 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e27f87-bc58-4707-96d4-d653f96ac613 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:26.422222 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e27f87-bc58-4707-96d4-d653f96ac613 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:26.422222 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e27f87-bc58-4707-96d4-d653f96ac613 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:26.441205 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 946/1859] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:26 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:47:26.446352 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-680c1600-86e1-4ade-be21-47911511ed20 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:26.493013 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:26.503862 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37e27f87-bc58-4707-96d4-d653f96ac613 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:26.505609 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37e27f87-bc58-4707-96d4-d653f96ac613 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:26.516125 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 914/1860] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:26 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 243 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 15:47:26.516443 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-680c1600-86e1-4ade-be21-47911511ed20 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:26.517431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df92b58-918b-4a09-883e-347546729a5e None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:26.517723 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df92b58-918b-4a09-883e-347546729a5e None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:26.517997 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df92b58-918b-4a09-883e-347546729a5e None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:26.519459 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 915/1861] 10.222.0.102 () {54 vars in 926 bytes} [Wed Aug 24 15:47:26 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:47:26.521535 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99bb57d8-385f-4905-94c7-d3a3011ab51c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:26.522428 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bb57d8-385f-4905-94c7-d3a3011ab51c admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:26.522602 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bb57d8-385f-4905-94c7-d3a3011ab51c admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:26.522884 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bb57d8-385f-4905-94c7-d3a3011ab51c admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:26.523884 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99bb57d8-385f-4905-94c7-d3a3011ab51c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:26.526037 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99bb57d8-385f-4905-94c7-d3a3011ab51c admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:26.535517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c0d30d-ed19-4c7c-8bcf-0087ff4559fc None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:26.536283 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c0d30d-ed19-4c7c-8bcf-0087ff4559fc None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:26.536607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c0d30d-ed19-4c7c-8bcf-0087ff4559fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:26.562751 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 947/1862] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:47:26 2022] POST /identity/v3/projects => generated 368 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 15:47:26.568021 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99bb57d8-385f-4905-94c7-d3a3011ab51c admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:26.568610 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-d5c0d30d-ed19-4c7c-8bcf-0087ff4559fc None None] MFA Rules not processed for user `0ee0002a76c3471f819b37fef510ef99`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:26.570662 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-d5c0d30d-ed19-4c7c-8bcf-0087ff4559fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:26.596848 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:26.738163 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99bb57d8-385f-4905-94c7-d3a3011ab51c admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:26.743906 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 916/1863] 10.222.0.102 () {58 vars in 1023 bytes} [Wed Aug 24 15:47:26 2022] POST /identity/v3/auth/tokens => generated 3318 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:26.744811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c0d30d-ed19-4c7c-8bcf-0087ff4559fc None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:26.746264 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-d5c0d30d-ed19-4c7c-8bcf-0087ff4559fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:26.753886 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a5f520-afc3-4dad-ae4b-6769506ef987 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:26.754521 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a5f520-afc3-4dad-ae4b-6769506ef987 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:26.764203 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a5f520-afc3-4dad-ae4b-6769506ef987 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:26.764647 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a5f520-afc3-4dad-ae4b-6769506ef987 admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:26.767533 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a5f520-afc3-4dad-ae4b-6769506ef987 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:26.772061 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a5f520-afc3-4dad-ae4b-6769506ef987 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:26.779164 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c0d30d-ed19-4c7c-8bcf-0087ff4559fc None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:26.785260 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dced163-b0d5-4abd-ab60-d0012408b99d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:26.785856 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dced163-b0d5-4abd-ab60-d0012408b99d admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:26.786166 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dced163-b0d5-4abd-ab60-d0012408b99d admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:26.786507 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dced163-b0d5-4abd-ab60-d0012408b99d admin admin] PATH_INFO: `/v3/users/7999201deb0d4f0391df21fdd0cdb60b` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:26.788000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dced163-b0d5-4abd-ab60-d0012408b99d admin admin] RBAC: Authorizing `identity:delete_user(user_id=7999201deb0d4f0391df21fdd0cdb60b)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:26.790424 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dced163-b0d5-4abd-ab60-d0012408b99d admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:26.823714 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 948/1864] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:47:26 2022] POST /identity/v3/users => generated 291 bytes in 257 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:26.824671 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a5f520-afc3-4dad-ae4b-6769506ef987 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:26.826681 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-78a5f520-afc3-4dad-ae4b-6769506ef987 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:26.844891 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a5f520-afc3-4dad-ae4b-6769506ef987 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:26.854835 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e773c47e-c017-430b-a073-9c1c524cafeb service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0ee0002a76c3471f819b37fef510ef99', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:26.855032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e773c47e-c017-430b-a073-9c1c524cafeb service placement] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:26.856285 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e773c47e-c017-430b-a073-9c1c524cafeb service placement] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:26.856285 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e773c47e-c017-430b-a073-9c1c524cafeb service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:26.868160 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e773c47e-c017-430b-a073-9c1c524cafeb service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:26.870100 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e773c47e-c017-430b-a073-9c1c524cafeb service placement] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:26.883399 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 949/1865] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:26 2022] GET /identity/v3/auth/tokens => generated 3318 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:26.884789 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e773c47e-c017-430b-a073-9c1c524cafeb service placement] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:26.900585 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e773c47e-c017-430b-a073-9c1c524cafeb service placement] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:26.902238 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-0dced163-b0d5-4abd-ab60-d0012408b99d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7999201deb0d4f0391df21fdd0cdb60b'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:26.903016 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-0dced163-b0d5-4abd-ab60-d0012408b99d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7999201deb0d4f0391df21fdd0cdb60b'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:26.908631 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b895d8-bc9a-4d37-8fa6-7d5a5ee3c5d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:26.910706 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b895d8-bc9a-4d37-8fa6-7d5a5ee3c5d8 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:26.911208 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b895d8-bc9a-4d37-8fa6-7d5a5ee3c5d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:26.911719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b895d8-bc9a-4d37-8fa6-7d5a5ee3c5d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:26.912603 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b895d8-bc9a-4d37-8fa6-7d5a5ee3c5d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:26.915605 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-0dced163-b0d5-4abd-ab60-d0012408b99d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7999201deb0d4f0391df21fdd0cdb60b'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:26.917887 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b895d8-bc9a-4d37-8fa6-7d5a5ee3c5d8 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:26.933660 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 950/1866] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:26 2022] GET /identity/v3/roles => generated 1427 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:26.934262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-0dced163-b0d5-4abd-ab60-d0012408b99d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7999201deb0d4f0391df21fdd0cdb60b'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:26.935077 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-0dced163-b0d5-4abd-ab60-d0012408b99d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7999201deb0d4f0391df21fdd0cdb60b'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:26.937599 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b895d8-bc9a-4d37-8fa6-7d5a5ee3c5d8 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:26.939817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-e7b895d8-bc9a-4d37-8fa6-7d5a5ee3c5d8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:26.954914 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b895d8-bc9a-4d37-8fa6-7d5a5ee3c5d8 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:26.961480 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d213189-8d85-429d-91da-511bf864f907 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:26.962449 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d213189-8d85-429d-91da-511bf864f907 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:26.962831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d213189-8d85-429d-91da-511bf864f907 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:26.963203 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d213189-8d85-429d-91da-511bf864f907 admin admin] PATH_INFO: `/v3/projects/f0a91faefeb54ee5a64cadeed8d8a524/users/3931c15eb05e4a3185e7a4dee7d21a52/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:26.963872 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 917/1867] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:47:26 2022] DELETE /identity/v3/users/7999201deb0d4f0391df21fdd0cdb60b => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:26.983415 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d213189-8d85-429d-91da-511bf864f907 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f0a91faefeb54ee5a64cadeed8d8a524, user_id=3931c15eb05e4a3185e7a4dee7d21a52, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:26.986797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d213189-8d85-429d-91da-511bf864f907 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:27.005216 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 951/1868] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:47:26 2022] PUT /identity/v3/projects/f0a91faefeb54ee5a64cadeed8d8a524/users/3931c15eb05e4a3185e7a4dee7d21a52/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:27.011156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dced163-b0d5-4abd-ab60-d0012408b99d admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:27.107330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dced163-b0d5-4abd-ab60-d0012408b99d admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:27.154574 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:27.189837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d213189-8d85-429d-91da-511bf864f907 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:27.191880 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-1d213189-8d85-429d-91da-511bf864f907 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:27.248988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d54dae-c0e3-47a2-ba5a-80bf5652d6fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:27.250876 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d54dae-c0e3-47a2-ba5a-80bf5652d6fd admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:27.250876 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d54dae-c0e3-47a2-ba5a-80bf5652d6fd admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:27.250876 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d54dae-c0e3-47a2-ba5a-80bf5652d6fd admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:27.251113 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d54dae-c0e3-47a2-ba5a-80bf5652d6fd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:27.253066 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d54dae-c0e3-47a2-ba5a-80bf5652d6fd admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:27.280668 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 918/1869] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:27 2022] GET /identity/v3/roles => generated 1427 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:27.281685 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d54dae-c0e3-47a2-ba5a-80bf5652d6fd admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:27.310508 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d54dae-c0e3-47a2-ba5a-80bf5652d6fd admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:27.324220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b92dc9e-463c-4582-be0b-85cb561d8749 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:27.325164 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b92dc9e-463c-4582-be0b-85cb561d8749 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:27.325520 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b92dc9e-463c-4582-be0b-85cb561d8749 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:27.325917 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b92dc9e-463c-4582-be0b-85cb561d8749 admin admin] PATH_INFO: `/v3/projects/f0a91faefeb54ee5a64cadeed8d8a524/users/3931c15eb05e4a3185e7a4dee7d21a52/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:27.329628 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b92dc9e-463c-4582-be0b-85cb561d8749 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f0a91faefeb54ee5a64cadeed8d8a524, user_id=3931c15eb05e4a3185e7a4dee7d21a52, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:27.333784 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b92dc9e-463c-4582-be0b-85cb561d8749 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:27.366207 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 919/1870] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:47:27 2022] PUT /identity/v3/projects/f0a91faefeb54ee5a64cadeed8d8a524/users/3931c15eb05e4a3185e7a4dee7d21a52/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:27.384589 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ed2618-1a2e-41e9-a376-697361bce17e None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:27.384878 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ed2618-1a2e-41e9-a376-697361bce17e None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:27.385153 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ed2618-1a2e-41e9-a376-697361bce17e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:27.402928 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d213189-8d85-429d-91da-511bf864f907 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:27.424652 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:27.474465 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-e5ed2618-1a2e-41e9-a376-697361bce17e None None] MFA Rules not processed for user `3931c15eb05e4a3185e7a4dee7d21a52`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:27.476079 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-e5ed2618-1a2e-41e9-a376-697361bce17e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:27.557716 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d62475-bcaa-4cd2-9b62-4a9f63641c2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:27.558708 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d62475-bcaa-4cd2-9b62-4a9f63641c2b admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:27.558990 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d62475-bcaa-4cd2-9b62-4a9f63641c2b admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:27.559268 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d62475-bcaa-4cd2-9b62-4a9f63641c2b admin admin] PATH_INFO: `/v3/projects/f2ed6eaddf0f445ab4437da183776596` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:27.560577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d62475-bcaa-4cd2-9b62-4a9f63641c2b admin admin] RBAC: Authorizing `identity:delete_project(project_id=f2ed6eaddf0f445ab4437da183776596)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:27.562718 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d62475-bcaa-4cd2-9b62-4a9f63641c2b admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:27.641874 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-f4d62475-bcaa-4cd2-9b62-4a9f63641c2b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f2ed6eaddf0f445ab4437da183776596', 'request_id': 'req-f4d62475-bcaa-4cd2-9b62-4a9f63641c2b'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:27.651936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-f4d62475-bcaa-4cd2-9b62-4a9f63641c2b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f2ed6eaddf0f445ab4437da183776596', 'request_id': 'req-f4d62475-bcaa-4cd2-9b62-4a9f63641c2b'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:27.670335 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-f4d62475-bcaa-4cd2-9b62-4a9f63641c2b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f2ed6eaddf0f445ab4437da183776596', 'request_id': 'req-f4d62475-bcaa-4cd2-9b62-4a9f63641c2b'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:27.677266 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 920/1871] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:27 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 295 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:27.678547 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ed2618-1a2e-41e9-a376-697361bce17e None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:27.693012 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-e5ed2618-1a2e-41e9-a376-697361bce17e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:27.697931 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-f4d62475-bcaa-4cd2-9b62-4a9f63641c2b admin admin] The token cache is being invalidate because project f2ed6eaddf0f445ab4437da183776596 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:47:27.698251 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-f4d62475-bcaa-4cd2-9b62-4a9f63641c2b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:27.700119 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-f4d62475-bcaa-4cd2-9b62-4a9f63641c2b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:27.709010 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 952/1872] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:47:27 2022] DELETE /identity/v3/projects/f2ed6eaddf0f445ab4437da183776596 => generated 0 bytes in 515 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:27.709232 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ed2618-1a2e-41e9-a376-697361bce17e None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:27.710083 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d62475-bcaa-4cd2-9b62-4a9f63641c2b admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:27.711369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-f4d62475-bcaa-4cd2-9b62-4a9f63641c2b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:27.811732 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f4e69a-3469-4144-9961-87a54205ea28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:27.812395 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f4e69a-3469-4144-9961-87a54205ea28 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:27.812676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f4e69a-3469-4144-9961-87a54205ea28 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:27.813313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f4e69a-3469-4144-9961-87a54205ea28 admin admin] PATH_INFO: `/v3/projects/ae84f1465e1444a5afde3ad7883cef33` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:27.814746 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f4e69a-3469-4144-9961-87a54205ea28 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ae84f1465e1444a5afde3ad7883cef33)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:27.817949 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f4e69a-3469-4144-9961-87a54205ea28 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:27.836609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d62475-bcaa-4cd2-9b62-4a9f63641c2b admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:27.842731 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f96db2b-1bbd-4745-91bd-b638a139d903 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:27.843369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f96db2b-1bbd-4745-91bd-b638a139d903 service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:27.844031 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f96db2b-1bbd-4745-91bd-b638a139d903 service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:27.844315 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f96db2b-1bbd-4745-91bd-b638a139d903 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:27.846265 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-1f96db2b-1bbd-4745-91bd-b638a139d903 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:27.930442 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:27.933878 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-c5f4e69a-3469-4144-9961-87a54205ea28 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ae84f1465e1444a5afde3ad7883cef33', 'request_id': 'req-c5f4e69a-3469-4144-9961-87a54205ea28'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:27.942596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-c5f4e69a-3469-4144-9961-87a54205ea28 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ae84f1465e1444a5afde3ad7883cef33', 'request_id': 'req-c5f4e69a-3469-4144-9961-87a54205ea28'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:27.949994 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-c5f4e69a-3469-4144-9961-87a54205ea28 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ae84f1465e1444a5afde3ad7883cef33', 'request_id': 'req-c5f4e69a-3469-4144-9961-87a54205ea28'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:27.952894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:27.971507 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f96db2b-1bbd-4745-91bd-b638a139d903 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:27.973281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-c5f4e69a-3469-4144-9961-87a54205ea28 admin admin] The token cache is being invalidate because project ae84f1465e1444a5afde3ad7883cef33 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:47:27.973745 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-c5f4e69a-3469-4144-9961-87a54205ea28 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:27.974321 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f96db2b-1bbd-4745-91bd-b638a139d903 service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:27.974825 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-c5f4e69a-3469-4144-9961-87a54205ea28 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:27.980740 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 921/1873] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:47:27 2022] DELETE /identity/v3/projects/ae84f1465e1444a5afde3ad7883cef33 => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:27.982384 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b52b03-bbf1-47d5-8c21-0e26796da150 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:27.982673 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b52b03-bbf1-47d5-8c21-0e26796da150 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:27.982951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b52b03-bbf1-47d5-8c21-0e26796da150 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:27.999909 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 953/1874] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:27 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:28.002450 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f96db2b-1bbd-4745-91bd-b638a139d903 service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:28.006073 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-1f96db2b-1bbd-4745-91bd-b638a139d903 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:28.013988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-39b52b03-bbf1-47d5-8c21-0e26796da150 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:28.015574 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-39b52b03-bbf1-47d5-8c21-0e26796da150 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:28.033000 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f96db2b-1bbd-4745-91bd-b638a139d903 service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:28.121885 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 922/1875] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:27 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:28.122887 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b52b03-bbf1-47d5-8c21-0e26796da150 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:28.124646 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-39b52b03-bbf1-47d5-8c21-0e26796da150 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:28.140070 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b52b03-bbf1-47d5-8c21-0e26796da150 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:28.146994 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f080779-e87f-4ecf-84f7-d4c7a1a68b49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:28.147964 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f080779-e87f-4ecf-84f7-d4c7a1a68b49 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:28.148314 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f080779-e87f-4ecf-84f7-d4c7a1a68b49 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:28.149402 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f080779-e87f-4ecf-84f7-d4c7a1a68b49 admin admin] PATH_INFO: `/v3/users/31ccfb389bc8400e85421f68d3608145` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:28.151597 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f080779-e87f-4ecf-84f7-d4c7a1a68b49 admin admin] RBAC: Authorizing `identity:delete_user(user_id=31ccfb389bc8400e85421f68d3608145)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:28.153342 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e7f31e-a17e-4451-957b-a47360d5798c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:28.153792 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f080779-e87f-4ecf-84f7-d4c7a1a68b49 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:28.154422 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e7f31e-a17e-4451-957b-a47360d5798c admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:28.154901 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e7f31e-a17e-4451-957b-a47360d5798c admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:28.155358 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e7f31e-a17e-4451-957b-a47360d5798c admin admin] PATH_INFO: `/v3/projects/f78537131d144cbebdf40cd763440080` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:28.157576 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e7f31e-a17e-4451-957b-a47360d5798c admin admin] RBAC: Authorizing `identity:delete_project(project_id=f78537131d144cbebdf40cd763440080)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:28.159837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e7f31e-a17e-4451-957b-a47360d5798c admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:28.241712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-d4e7f31e-a17e-4451-957b-a47360d5798c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f78537131d144cbebdf40cd763440080', 'request_id': 'req-d4e7f31e-a17e-4451-957b-a47360d5798c'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:28.258868 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-0f080779-e87f-4ecf-84f7-d4c7a1a68b49 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '31ccfb389bc8400e85421f68d3608145'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:28.259575 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-d4e7f31e-a17e-4451-957b-a47360d5798c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f78537131d144cbebdf40cd763440080', 'request_id': 'req-d4e7f31e-a17e-4451-957b-a47360d5798c'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:28.259836 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-0f080779-e87f-4ecf-84f7-d4c7a1a68b49 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '31ccfb389bc8400e85421f68d3608145'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:28.270483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-0f080779-e87f-4ecf-84f7-d4c7a1a68b49 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '31ccfb389bc8400e85421f68d3608145'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:28.271702 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-d4e7f31e-a17e-4451-957b-a47360d5798c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f78537131d144cbebdf40cd763440080', 'request_id': 'req-d4e7f31e-a17e-4451-957b-a47360d5798c'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:28.289672 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-0f080779-e87f-4ecf-84f7-d4c7a1a68b49 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '31ccfb389bc8400e85421f68d3608145'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:28.290503 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-0f080779-e87f-4ecf-84f7-d4c7a1a68b49 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '31ccfb389bc8400e85421f68d3608145'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:28.309100 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-d4e7f31e-a17e-4451-957b-a47360d5798c admin admin] The token cache is being invalidate because project f78537131d144cbebdf40cd763440080 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:47:28.311029 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-d4e7f31e-a17e-4451-957b-a47360d5798c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:28.312510 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-d4e7f31e-a17e-4451-957b-a47360d5798c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:28.315362 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 954/1876] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:47:27 2022] DELETE /identity/v3/projects/f78537131d144cbebdf40cd763440080 => generated 0 bytes in 315 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:28.318494 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e7f31e-a17e-4451-957b-a47360d5798c admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:28.324279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-d4e7f31e-a17e-4451-957b-a47360d5798c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:28.324484 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 923/1877] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:47:28 2022] DELETE /identity/v3/users/31ccfb389bc8400e85421f68d3608145 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:28.327454 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b5a963-52d7-4b19-a321-b8cbc93d6140 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:28.327898 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b5a963-52d7-4b19-a321-b8cbc93d6140 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:28.330090 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b5a963-52d7-4b19-a321-b8cbc93d6140 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:28.383149 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-e7b5a963-52d7-4b19-a321-b8cbc93d6140 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:28.386255 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-e7b5a963-52d7-4b19-a321-b8cbc93d6140 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:28.405251 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e7f31e-a17e-4451-957b-a47360d5798c admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:28.457469 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:28.462385 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:28.486049 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47f4dc4-6672-461d-84ff-99410e762df9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:28.487238 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47f4dc4-6672-461d-84ff-99410e762df9 service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:28.488010 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47f4dc4-6672-461d-84ff-99410e762df9 service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:28.488447 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47f4dc4-6672-461d-84ff-99410e762df9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:28.497552 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 924/1878] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:28 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:28.497961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-e47f4dc4-6672-461d-84ff-99410e762df9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:28.498661 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b5a963-52d7-4b19-a321-b8cbc93d6140 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:28.500424 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-e7b5a963-52d7-4b19-a321-b8cbc93d6140 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:28.517831 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b5a963-52d7-4b19-a321-b8cbc93d6140 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:28.523208 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fcf0593-fddc-488d-ac97-f699f4565ab1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:28.524591 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fcf0593-fddc-488d-ac97-f699f4565ab1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:28.524991 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fcf0593-fddc-488d-ac97-f699f4565ab1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:28.525283 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fcf0593-fddc-488d-ac97-f699f4565ab1 admin admin] PATH_INFO: `/v3/users/c1adb6fdaabe4e2296dfcd5ccc117662` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:28.526801 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fcf0593-fddc-488d-ac97-f699f4565ab1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c1adb6fdaabe4e2296dfcd5ccc117662)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:28.529393 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fcf0593-fddc-488d-ac97-f699f4565ab1 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:28.589172 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e47f4dc4-6672-461d-84ff-99410e762df9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:28.590674 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e47f4dc4-6672-461d-84ff-99410e762df9 service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:28.606636 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 955/1879] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:28 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 289 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:28.608344 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-1fcf0593-fddc-488d-ac97-f699f4565ab1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c1adb6fdaabe4e2296dfcd5ccc117662'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:28.609038 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-1fcf0593-fddc-488d-ac97-f699f4565ab1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c1adb6fdaabe4e2296dfcd5ccc117662'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:28.618458 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-1fcf0593-fddc-488d-ac97-f699f4565ab1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c1adb6fdaabe4e2296dfcd5ccc117662'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:28.619770 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47f4dc4-6672-461d-84ff-99410e762df9 service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:28.643758 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-1fcf0593-fddc-488d-ac97-f699f4565ab1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c1adb6fdaabe4e2296dfcd5ccc117662'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:28.643758 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-1fcf0593-fddc-488d-ac97-f699f4565ab1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c1adb6fdaabe4e2296dfcd5ccc117662'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:28.657934 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 925/1880] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:47:28 2022] DELETE /identity/v3/users/c1adb6fdaabe4e2296dfcd5ccc117662 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:28.658562 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47f4dc4-6672-461d-84ff-99410e762df9 service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:28.770759 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c814e1-b058-448c-b6f6-4540b069eb8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:28.771612 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c814e1-b058-448c-b6f6-4540b069eb8e admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:28.771755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c814e1-b058-448c-b6f6-4540b069eb8e admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:28.772029 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c814e1-b058-448c-b6f6-4540b069eb8e admin admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:28.772747 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c814e1-b058-448c-b6f6-4540b069eb8e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:47:28.773064 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c814e1-b058-448c-b6f6-4540b069eb8e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:28.774969 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c814e1-b058-448c-b6f6-4540b069eb8e admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:28.796174 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 956/1881] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:47:28 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:47:28.797626 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fcf0593-fddc-488d-ac97-f699f4565ab1 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:28.800204 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-1fcf0593-fddc-488d-ac97-f699f4565ab1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:28.838450 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fcf0593-fddc-488d-ac97-f699f4565ab1 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:28.851960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a9b5ee-2339-45db-9d40-ba0a8e0127f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:28.852808 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a9b5ee-2339-45db-9d40-ba0a8e0127f3 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:28.853083 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a9b5ee-2339-45db-9d40-ba0a8e0127f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:28.853362 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a9b5ee-2339-45db-9d40-ba0a8e0127f3 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:28.854023 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a9b5ee-2339-45db-9d40-ba0a8e0127f3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:28.856009 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a9b5ee-2339-45db-9d40-ba0a8e0127f3 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:28.912874 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 926/1882] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:47:28 2022] POST /identity/v3/projects => generated 378 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:28.918190 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c814e1-b058-448c-b6f6-4540b069eb8e admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:28.934431 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c814e1-b058-448c-b6f6-4540b069eb8e admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:29.034393 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97e38d8-8ef0-409f-8cc5-7ea22a010786 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:29.035494 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97e38d8-8ef0-409f-8cc5-7ea22a010786 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:29.036082 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97e38d8-8ef0-409f-8cc5-7ea22a010786 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:29.036459 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97e38d8-8ef0-409f-8cc5-7ea22a010786 admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:29.037400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97e38d8-8ef0-409f-8cc5-7ea22a010786 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:29.041137 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97e38d8-8ef0-409f-8cc5-7ea22a010786 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:29.111500 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 957/1883] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:47:28 2022] POST /identity/v3/users => generated 290 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 15:47:29.113835 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a9b5ee-2339-45db-9d40-ba0a8e0127f3 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:29.147982 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a9b5ee-2339-45db-9d40-ba0a8e0127f3 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:29.168486 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff35302-a929-49a2-98b7-5882391ffe45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:29.171437 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff35302-a929-49a2-98b7-5882391ffe45 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:29.171756 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff35302-a929-49a2-98b7-5882391ffe45 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:29.173912 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff35302-a929-49a2-98b7-5882391ffe45 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:29.174413 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff35302-a929-49a2-98b7-5882391ffe45 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:29.184835 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff35302-a929-49a2-98b7-5882391ffe45 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:29.232517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 927/1884] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:29 2022] GET /identity/v3/roles => generated 1427 bytes in 119 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:29.234838 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97e38d8-8ef0-409f-8cc5-7ea22a010786 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:29.267313 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97e38d8-8ef0-409f-8cc5-7ea22a010786 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:29.283133 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56142c5d-94ef-45b8-bcd0-c16ffb6159ad None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:29.283323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56142c5d-94ef-45b8-bcd0-c16ffb6159ad None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:29.284470 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56142c5d-94ef-45b8-bcd0-c16ffb6159ad None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:29.286877 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 928/1885] 10.222.0.102 () {54 vars in 926 bytes} [Wed Aug 24 15:47:29 2022] GET /identity => generated 271 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:47:29.289945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805daa1f-2ff1-480a-b442-8e2329aff601 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:29.300069 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805daa1f-2ff1-480a-b442-8e2329aff601 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:29.300543 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805daa1f-2ff1-480a-b442-8e2329aff601 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:29.300843 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805daa1f-2ff1-480a-b442-8e2329aff601 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:29.302047 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805daa1f-2ff1-480a-b442-8e2329aff601 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:29.310588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805daa1f-2ff1-480a-b442-8e2329aff601 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:29.339717 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b0baa9-d664-4d5d-b540-cbfb3e81fd28 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:29.340067 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b0baa9-d664-4d5d-b540-cbfb3e81fd28 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:29.340413 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b0baa9-d664-4d5d-b540-cbfb3e81fd28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:29.347674 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 958/1886] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:29 2022] GET /identity/v3/roles => generated 1427 bytes in 112 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:29.353225 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805daa1f-2ff1-480a-b442-8e2329aff601 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:29.370962 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:29.376325 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805daa1f-2ff1-480a-b442-8e2329aff601 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:29.391254 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b09c108-42ae-41d4-b2fb-4eac40f760bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:29.391974 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b09c108-42ae-41d4-b2fb-4eac40f760bc admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:29.392260 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b09c108-42ae-41d4-b2fb-4eac40f760bc admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:29.392546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b09c108-42ae-41d4-b2fb-4eac40f760bc admin admin] PATH_INFO: `/v3/projects/6d781073258242179398fe7e35565d37/users/4881ba284a8e481bab0452ab8527c0aa/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:29.395693 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-51b0baa9-d664-4d5d-b540-cbfb3e81fd28 None None] MFA Rules not processed for user `e0d544379fe44438ad10289b45124374`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:29.398918 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-51b0baa9-d664-4d5d-b540-cbfb3e81fd28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:29.421991 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b09c108-42ae-41d4-b2fb-4eac40f760bc admin admin] RBAC: Authorizing `identity:create_grant(project_id=6d781073258242179398fe7e35565d37, user_id=4881ba284a8e481bab0452ab8527c0aa, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:29.426400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b09c108-42ae-41d4-b2fb-4eac40f760bc admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:29.455360 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 959/1887] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:47:29 2022] PUT /identity/v3/projects/6d781073258242179398fe7e35565d37/users/4881ba284a8e481bab0452ab8527c0aa/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:29.456437 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b09c108-42ae-41d4-b2fb-4eac40f760bc admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:29.458691 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-8b09c108-42ae-41d4-b2fb-4eac40f760bc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:29.540131 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b09c108-42ae-41d4-b2fb-4eac40f760bc admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:29.543003 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 929/1888] 10.222.0.102 () {58 vars in 1023 bytes} [Wed Aug 24 15:47:29 2022] POST /identity/v3/auth/tokens => generated 3382 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:47:29.544601 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef0fe7e-0ca0-4084-9d2f-a573bc86d0e5 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:29.544967 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef0fe7e-0ca0-4084-9d2f-a573bc86d0e5 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:29.545159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef0fe7e-0ca0-4084-9d2f-a573bc86d0e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:29.551373 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55ba1f2-6a97-4e2e-9824-090878e9de45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:29.552057 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55ba1f2-6a97-4e2e-9824-090878e9de45 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:29.552527 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55ba1f2-6a97-4e2e-9824-090878e9de45 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:29.552646 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55ba1f2-6a97-4e2e-9824-090878e9de45 admin admin] PATH_INFO: `/v3/projects/3df5583daab94bcd8fdc3806b48645d1` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:29.553728 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c55ba1f2-6a97-4e2e-9824-090878e9de45 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3df5583daab94bcd8fdc3806b48645d1)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:29.556651 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c55ba1f2-6a97-4e2e-9824-090878e9de45 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:29.578016 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-aef0fe7e-0ca0-4084-9d2f-a573bc86d0e5 None None] MFA Rules not processed for user `4881ba284a8e481bab0452ab8527c0aa`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:29.579976 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-aef0fe7e-0ca0-4084-9d2f-a573bc86d0e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:29.585582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:29.614224 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-c55ba1f2-6a97-4e2e-9824-090878e9de45 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3df5583daab94bcd8fdc3806b48645d1', 'request_id': 'req-c55ba1f2-6a97-4e2e-9824-090878e9de45'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:29.623131 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-c55ba1f2-6a97-4e2e-9824-090878e9de45 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3df5583daab94bcd8fdc3806b48645d1', 'request_id': 'req-c55ba1f2-6a97-4e2e-9824-090878e9de45'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:29.629881 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-c55ba1f2-6a97-4e2e-9824-090878e9de45 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3df5583daab94bcd8fdc3806b48645d1', 'request_id': 'req-c55ba1f2-6a97-4e2e-9824-090878e9de45'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:29.646381 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-c55ba1f2-6a97-4e2e-9824-090878e9de45 admin admin] The token cache is being invalidate because project 3df5583daab94bcd8fdc3806b48645d1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:47:29.646681 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-c55ba1f2-6a97-4e2e-9824-090878e9de45 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:29.647581 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-c55ba1f2-6a97-4e2e-9824-090878e9de45 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:29.649206 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 960/1889] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:47:29 2022] DELETE /identity/v3/projects/3df5583daab94bcd8fdc3806b48645d1 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:29.650195 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55ba1f2-6a97-4e2e-9824-090878e9de45 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:29.651582 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-c55ba1f2-6a97-4e2e-9824-090878e9de45 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:29.670772 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 930/1890] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:29 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:29.671595 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aef0fe7e-0ca0-4084-9d2f-a573bc86d0e5 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:29.673094 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-aef0fe7e-0ca0-4084-9d2f-a573bc86d0e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:29.735956 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55ba1f2-6a97-4e2e-9824-090878e9de45 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:29.740675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd86180b-8219-44e1-bd56-fa402921d7b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:29.741307 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd86180b-8219-44e1-bd56-fa402921d7b5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:29.741592 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd86180b-8219-44e1-bd56-fa402921d7b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:29.741863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd86180b-8219-44e1-bd56-fa402921d7b5 admin admin] PATH_INFO: `/v3/projects/188a0715b8094f30a3de10f4b410d397` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:29.743907 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd86180b-8219-44e1-bd56-fa402921d7b5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=188a0715b8094f30a3de10f4b410d397)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:29.745953 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd86180b-8219-44e1-bd56-fa402921d7b5 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:29.761153 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aef0fe7e-0ca0-4084-9d2f-a573bc86d0e5 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:29.763269 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:29.814503 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-bd86180b-8219-44e1-bd56-fa402921d7b5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '188a0715b8094f30a3de10f4b410d397', 'request_id': 'req-bd86180b-8219-44e1-bd56-fa402921d7b5'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:29.833072 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-bd86180b-8219-44e1-bd56-fa402921d7b5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '188a0715b8094f30a3de10f4b410d397', 'request_id': 'req-bd86180b-8219-44e1-bd56-fa402921d7b5'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:29.857646 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-bd86180b-8219-44e1-bd56-fa402921d7b5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '188a0715b8094f30a3de10f4b410d397', 'request_id': 'req-bd86180b-8219-44e1-bd56-fa402921d7b5'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:29.881977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-bd86180b-8219-44e1-bd56-fa402921d7b5 admin admin] The token cache is being invalidate because project 188a0715b8094f30a3de10f4b410d397 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:47:29.882302 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-bd86180b-8219-44e1-bd56-fa402921d7b5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:29.884055 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-bd86180b-8219-44e1-bd56-fa402921d7b5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:29.885025 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7320672-d646-4217-9758-19841281ae68 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:29.885647 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7320672-d646-4217-9758-19841281ae68 service nova] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:29.886069 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7320672-d646-4217-9758-19841281ae68 service nova] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:29.886181 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7320672-d646-4217-9758-19841281ae68 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:29.888027 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 961/1891] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:47:29 2022] DELETE /identity/v3/projects/188a0715b8094f30a3de10f4b410d397 => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:29.889312 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a363022-ac2a-4fe0-99df-cff3d6bc9f79 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:29.889591 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a363022-ac2a-4fe0-99df-cff3d6bc9f79 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:29.889863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a363022-ac2a-4fe0-99df-cff3d6bc9f79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:29.894192 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-e7320672-d646-4217-9758-19841281ae68 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:29.914605 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7320672-d646-4217-9758-19841281ae68 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:29.917840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7320672-d646-4217-9758-19841281ae68 service nova] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:29.932542 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 931/1892] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:29 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 261 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:29.932764 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-4a363022-ac2a-4fe0-99df-cff3d6bc9f79 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:29.933878 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7320672-d646-4217-9758-19841281ae68 service nova] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:29.934349 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-4a363022-ac2a-4fe0-99df-cff3d6bc9f79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:29.936103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-e7320672-d646-4217-9758-19841281ae68 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:29.957607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7320672-d646-4217-9758-19841281ae68 service nova] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:30.001513 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:30.029976 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 962/1893] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:29 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:30.032439 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d179d80a-bbfd-4083-855e-11839a9a7d9c None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:30.032876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d179d80a-bbfd-4083-855e-11839a9a7d9c None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:30.033305 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d179d80a-bbfd-4083-855e-11839a9a7d9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:30.068881 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4014466-6080-4e5d-919f-5c4717d419f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:30.069703 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4014466-6080-4e5d-919f-5c4717d419f8 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:30.070275 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4014466-6080-4e5d-919f-5c4717d419f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:30.070755 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4014466-6080-4e5d-919f-5c4717d419f8 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:30.071230 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-d179d80a-bbfd-4083-855e-11839a9a7d9c None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:30.071876 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4014466-6080-4e5d-919f-5c4717d419f8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:30.072512 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-d179d80a-bbfd-4083-855e-11839a9a7d9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:30.074515 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4014466-6080-4e5d-919f-5c4717d419f8 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:30.095672 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 963/1894] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:30 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 65 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:30.097180 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89abe2d-31b6-495c-b8ec-419b85ae1f85 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:30.097512 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89abe2d-31b6-495c-b8ec-419b85ae1f85 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:30.098941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89abe2d-31b6-495c-b8ec-419b85ae1f85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:30.116103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 932/1895] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:47:29 2022] POST /identity/v3/projects => generated 378 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:30.118002 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff21008c-1733-4050-b746-63a3d341c8b8 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:30.118301 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff21008c-1733-4050-b746-63a3d341c8b8 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:30.118581 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff21008c-1733-4050-b746-63a3d341c8b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:30.148716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-ff21008c-1733-4050-b746-63a3d341c8b8 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:30.150692 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-ff21008c-1733-4050-b746-63a3d341c8b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:30.152255 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-b89abe2d-31b6-495c-b8ec-419b85ae1f85 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:30.154039 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-b89abe2d-31b6-495c-b8ec-419b85ae1f85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:30.236149 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 933/1896] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:30 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:30.236149 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff21008c-1733-4050-b746-63a3d341c8b8 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:30.247233 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff21008c-1733-4050-b746-63a3d341c8b8 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:30.253431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f07415-b95a-4e14-a145-1febedf2220b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:30.254216 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f07415-b95a-4e14-a145-1febedf2220b admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:30.254504 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f07415-b95a-4e14-a145-1febedf2220b admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:30.254781 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f07415-b95a-4e14-a145-1febedf2220b admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:30.255433 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f07415-b95a-4e14-a145-1febedf2220b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:30.260125 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f07415-b95a-4e14-a145-1febedf2220b admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:30.284343 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 964/1897] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:30 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:30.285630 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89abe2d-31b6-495c-b8ec-419b85ae1f85 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:30.289606 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 934/1898] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:47:30 2022] POST /identity/v3/users => generated 296 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:30.291038 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f07415-b95a-4e14-a145-1febedf2220b admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:30.304321 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89abe2d-31b6-495c-b8ec-419b85ae1f85 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:30.310285 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d63e24-44eb-4ca6-86ec-f69a355da6db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:30.310905 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d63e24-44eb-4ca6-86ec-f69a355da6db admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:30.311184 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d63e24-44eb-4ca6-86ec-f69a355da6db admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:30.311456 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d63e24-44eb-4ca6-86ec-f69a355da6db admin admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:30.312141 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d63e24-44eb-4ca6-86ec-f69a355da6db admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:47:30.312448 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d63e24-44eb-4ca6-86ec-f69a355da6db admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:30.314080 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d63e24-44eb-4ca6-86ec-f69a355da6db admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:30.319463 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f07415-b95a-4e14-a145-1febedf2220b admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:30.335003 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1bd5b2-e296-494d-b0af-14921c57f33e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:30.337568 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1bd5b2-e296-494d-b0af-14921c57f33e admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:30.338149 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1bd5b2-e296-494d-b0af-14921c57f33e admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:30.340476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1bd5b2-e296-494d-b0af-14921c57f33e admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:30.341336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1bd5b2-e296-494d-b0af-14921c57f33e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:47:30.341754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1bd5b2-e296-494d-b0af-14921c57f33e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:30.344908 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 965/1899] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:47:30 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:47:30.345933 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d63e24-44eb-4ca6-86ec-f69a355da6db admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:30.346940 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1bd5b2-e296-494d-b0af-14921c57f33e admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:30.370189 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d63e24-44eb-4ca6-86ec-f69a355da6db admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:30.370754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 935/1900] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:47:30 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:47:30.371699 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1bd5b2-e296-494d-b0af-14921c57f33e admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:30.379194 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24d0d3a-7050-41e2-930a-84766e4eade7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:30.380019 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24d0d3a-7050-41e2-930a-84766e4eade7 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:30.380561 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24d0d3a-7050-41e2-930a-84766e4eade7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:30.381142 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24d0d3a-7050-41e2-930a-84766e4eade7 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:30.382413 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24d0d3a-7050-41e2-930a-84766e4eade7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:30.384531 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24d0d3a-7050-41e2-930a-84766e4eade7 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:30.397825 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1bd5b2-e296-494d-b0af-14921c57f33e admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:30.398718 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 966/1901] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:30 2022] GET /identity/v3/roles => generated 1427 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:30.400449 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24d0d3a-7050-41e2-930a-84766e4eade7 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:30.411376 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62202197-e1f6-427b-9012-efc827d238ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:30.412034 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62202197-e1f6-427b-9012-efc827d238ba admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:30.412313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62202197-e1f6-427b-9012-efc827d238ba admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:30.412965 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62202197-e1f6-427b-9012-efc827d238ba admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:30.415039 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62202197-e1f6-427b-9012-efc827d238ba admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:30.416373 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24d0d3a-7050-41e2-930a-84766e4eade7 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:30.421098 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62202197-e1f6-427b-9012-efc827d238ba admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:30.421263 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43390ebb-1de8-4b75-84b5-431706c36eb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:30.421793 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43390ebb-1de8-4b75-84b5-431706c36eb8 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:30.422076 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43390ebb-1de8-4b75-84b5-431706c36eb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:30.422625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43390ebb-1de8-4b75-84b5-431706c36eb8 admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:30.422974 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43390ebb-1de8-4b75-84b5-431706c36eb8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:30.424927 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43390ebb-1de8-4b75-84b5-431706c36eb8 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:30.437240 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:30.471671 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 967/1902] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:47:30 2022] POST /identity/v3/projects => generated 384 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:30.472450 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43390ebb-1de8-4b75-84b5-431706c36eb8 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:30.485938 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 936/1903] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:47:30 2022] POST /identity/v3/projects => generated 356 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:30.486760 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62202197-e1f6-427b-9012-efc827d238ba admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:30.576189 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43390ebb-1de8-4b75-84b5-431706c36eb8 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:30.583950 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07f74c2-6a42-4f33-ab59-d125f8027dfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:30.586912 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07f74c2-6a42-4f33-ab59-d125f8027dfa admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:30.587325 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62202197-e1f6-427b-9012-efc827d238ba admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:30.587448 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07f74c2-6a42-4f33-ab59-d125f8027dfa admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:30.587726 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07f74c2-6a42-4f33-ab59-d125f8027dfa admin admin] PATH_INFO: `/v3/projects/275cbfa2c18248f0b6d2289bd0906465/users/079bbec14aaf48678e62d829e31ffa67/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:30.595396 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361f0e89-59ff-4eff-9961-406636b361fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:30.596333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361f0e89-59ff-4eff-9961-406636b361fa admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:30.596728 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361f0e89-59ff-4eff-9961-406636b361fa admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:30.597842 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361f0e89-59ff-4eff-9961-406636b361fa admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:30.598721 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361f0e89-59ff-4eff-9961-406636b361fa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:30.602206 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361f0e89-59ff-4eff-9961-406636b361fa admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:30.611212 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07f74c2-6a42-4f33-ab59-d125f8027dfa admin admin] RBAC: Authorizing `identity:create_grant(project_id=275cbfa2c18248f0b6d2289bd0906465, user_id=079bbec14aaf48678e62d829e31ffa67, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:30.614227 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07f74c2-6a42-4f33-ab59-d125f8027dfa admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:30.619272 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:30.640034 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 968/1904] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:47:30 2022] PUT /identity/v3/projects/275cbfa2c18248f0b6d2289bd0906465/users/079bbec14aaf48678e62d829e31ffa67/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:30.640034 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07f74c2-6a42-4f33-ab59-d125f8027dfa admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:30.656924 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 937/1905] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:47:30 2022] POST /identity/v3/users => generated 293 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:30.658053 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361f0e89-59ff-4eff-9961-406636b361fa admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:30.705866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361f0e89-59ff-4eff-9961-406636b361fa admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:30.766988 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07f74c2-6a42-4f33-ab59-d125f8027dfa admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:30.774500 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef75516-ecf4-4b70-8c35-683fb3619fea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:30.775228 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef75516-ecf4-4b70-8c35-683fb3619fea admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:30.775515 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef75516-ecf4-4b70-8c35-683fb3619fea admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:30.775876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef75516-ecf4-4b70-8c35-683fb3619fea admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:30.776534 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ef75516-ecf4-4b70-8c35-683fb3619fea admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:30.779326 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ef75516-ecf4-4b70-8c35-683fb3619fea admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:30.820129 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 969/1906] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:47:30 2022] POST /identity/v3/users => generated 279 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:30.822877 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef75516-ecf4-4b70-8c35-683fb3619fea admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:30.836646 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165c39d5-66b6-4363-99c6-1bcddd2ea094 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:30.837463 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165c39d5-66b6-4363-99c6-1bcddd2ea094 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:30.837771 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165c39d5-66b6-4363-99c6-1bcddd2ea094 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:30.838047 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165c39d5-66b6-4363-99c6-1bcddd2ea094 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:30.838722 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-165c39d5-66b6-4363-99c6-1bcddd2ea094 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:30.841037 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-165c39d5-66b6-4363-99c6-1bcddd2ea094 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:30.844064 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef75516-ecf4-4b70-8c35-683fb3619fea admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:30.859575 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef98b663-be6b-47bb-ba4a-bc04e65c7aad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:30.860327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef98b663-be6b-47bb-ba4a-bc04e65c7aad admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:30.860664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef98b663-be6b-47bb-ba4a-bc04e65c7aad admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:30.861482 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef98b663-be6b-47bb-ba4a-bc04e65c7aad admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:30.862864 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef98b663-be6b-47bb-ba4a-bc04e65c7aad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:30.868418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 938/1907] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:30 2022] GET /identity/v3/roles => generated 1427 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:30.868418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165c39d5-66b6-4363-99c6-1bcddd2ea094 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:30.868789 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef98b663-be6b-47bb-ba4a-bc04e65c7aad admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:30.881489 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 970/1908] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:30 2022] GET /identity/v3/roles => generated 1427 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:30.882364 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef98b663-be6b-47bb-ba4a-bc04e65c7aad admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:30.888502 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165c39d5-66b6-4363-99c6-1bcddd2ea094 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:30.895546 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee2dade-58c6-4971-a145-a4215618e176 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:30.896275 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef98b663-be6b-47bb-ba4a-bc04e65c7aad admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:30.897852 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee2dade-58c6-4971-a145-a4215618e176 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:30.898168 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee2dade-58c6-4971-a145-a4215618e176 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:30.899195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee2dade-58c6-4971-a145-a4215618e176 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:30.900851 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee2dade-58c6-4971-a145-a4215618e176 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:30.903374 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee2dade-58c6-4971-a145-a4215618e176 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:30.903923 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f94fe3-2bdb-448e-a81a-9ff3ee463e69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:30.904693 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f94fe3-2bdb-448e-a81a-9ff3ee463e69 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:30.905068 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f94fe3-2bdb-448e-a81a-9ff3ee463e69 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:30.905442 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f94fe3-2bdb-448e-a81a-9ff3ee463e69 admin admin] PATH_INFO: `/v3/projects/275cbfa2c18248f0b6d2289bd0906465/users/079bbec14aaf48678e62d829e31ffa67/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:30.908317 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f94fe3-2bdb-448e-a81a-9ff3ee463e69 admin admin] RBAC: Authorizing `identity:create_grant(project_id=275cbfa2c18248f0b6d2289bd0906465, user_id=079bbec14aaf48678e62d829e31ffa67, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:30.914268 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f94fe3-2bdb-448e-a81a-9ff3ee463e69 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:30.925500 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 939/1909] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:30 2022] GET /identity/v3/roles => generated 1427 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:30.925500 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee2dade-58c6-4971-a145-a4215618e176 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:30.942327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 971/1910] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:47:30 2022] PUT /identity/v3/projects/275cbfa2c18248f0b6d2289bd0906465/users/079bbec14aaf48678e62d829e31ffa67/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:30.942327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f94fe3-2bdb-448e-a81a-9ff3ee463e69 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:30.962818 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee2dade-58c6-4971-a145-a4215618e176 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:31.007275 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:31.035728 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f94fe3-2bdb-448e-a81a-9ff3ee463e69 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:31.041778 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b902ace-896f-4ce1-bc8b-76a07a8e61df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:31.042554 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b902ace-896f-4ce1-bc8b-76a07a8e61df admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:31.043064 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b902ace-896f-4ce1-bc8b-76a07a8e61df admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:31.043438 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b902ace-896f-4ce1-bc8b-76a07a8e61df admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:31.044233 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b902ace-896f-4ce1-bc8b-76a07a8e61df admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:31.046114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b902ace-896f-4ce1-bc8b-76a07a8e61df admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:31.062229 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 972/1911] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:30 2022] GET /identity/v3/roles => generated 1427 bytes in 130 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:31.063810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d09b8a-5ee5-4c77-af3d-9dda992617b2 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:31.064213 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d09b8a-5ee5-4c77-af3d-9dda992617b2 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:31.064592 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d09b8a-5ee5-4c77-af3d-9dda992617b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:31.102633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-30d09b8a-5ee5-4c77-af3d-9dda992617b2 None None] MFA Rules not processed for user `079bbec14aaf48678e62d829e31ffa67`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:31.104165 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-30d09b8a-5ee5-4c77-af3d-9dda992617b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:31.107443 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b99f717-cdf5-49cf-bdbf-d351f5a96287 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:31.108256 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b99f717-cdf5-49cf-bdbf-d351f5a96287 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:31.108555 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b99f717-cdf5-49cf-bdbf-d351f5a96287 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:31.108971 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b99f717-cdf5-49cf-bdbf-d351f5a96287 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:31.109629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b99f717-cdf5-49cf-bdbf-d351f5a96287 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:31.111241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b99f717-cdf5-49cf-bdbf-d351f5a96287 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:31.139615 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 940/1912] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:30 2022] GET /identity/v3/roles => generated 1427 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:31.139615 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b99f717-cdf5-49cf-bdbf-d351f5a96287 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:31.166276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b99f717-cdf5-49cf-bdbf-d351f5a96287 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:31.190111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78564192-8601-49f2-ad69-9e7b57ee8d23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:31.190805 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78564192-8601-49f2-ad69-9e7b57ee8d23 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:31.194261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78564192-8601-49f2-ad69-9e7b57ee8d23 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:31.194578 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78564192-8601-49f2-ad69-9e7b57ee8d23 admin admin] PATH_INFO: `/v3/projects/9df6fd89c8854f55b0c274b7563b664b/users/28da55f3d8da4bc486c210bb198249f4/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:31.215445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 973/1913] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:31 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:31.216879 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d09b8a-5ee5-4c77-af3d-9dda992617b2 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:31.230270 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78564192-8601-49f2-ad69-9e7b57ee8d23 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9df6fd89c8854f55b0c274b7563b664b, user_id=28da55f3d8da4bc486c210bb198249f4, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:31.234245 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78564192-8601-49f2-ad69-9e7b57ee8d23 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:31.247117 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d09b8a-5ee5-4c77-af3d-9dda992617b2 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:31.253122 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 941/1914] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:47:31 2022] PUT /identity/v3/projects/9df6fd89c8854f55b0c274b7563b664b/users/28da55f3d8da4bc486c210bb198249f4/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:31.255846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78564192-8601-49f2-ad69-9e7b57ee8d23 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:31.258829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-78564192-8601-49f2-ad69-9e7b57ee8d23 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:31.380286 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78564192-8601-49f2-ad69-9e7b57ee8d23 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:31.390185 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2073c7f3-007c-4ba0-abc3-c35a7e2d2417 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:31.394466 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84649c9-a32e-4100-bfa3-d89600da1b74 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:31.395723 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2073c7f3-007c-4ba0-abc3-c35a7e2d2417 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:31.396044 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2073c7f3-007c-4ba0-abc3-c35a7e2d2417 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:31.396326 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2073c7f3-007c-4ba0-abc3-c35a7e2d2417 admin admin] PATH_INFO: `/v3/projects/cb0b0652e3df461093681ec09ffbf121/users/2b50e2d42f2541f1af0dd4f4e55fe7c1/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:31.397722 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84649c9-a32e-4100-bfa3-d89600da1b74 service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:31.398021 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84649c9-a32e-4100-bfa3-d89600da1b74 service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:31.401405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84649c9-a32e-4100-bfa3-d89600da1b74 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:31.409137 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-e84649c9-a32e-4100-bfa3-d89600da1b74 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:31.417139 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2073c7f3-007c-4ba0-abc3-c35a7e2d2417 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb0b0652e3df461093681ec09ffbf121, user_id=2b50e2d42f2541f1af0dd4f4e55fe7c1, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:31.418430 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:31.421129 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2073c7f3-007c-4ba0-abc3-c35a7e2d2417 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:31.449374 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 974/1915] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:47:31 2022] PUT /identity/v3/projects/cb0b0652e3df461093681ec09ffbf121/users/2b50e2d42f2541f1af0dd4f4e55fe7c1/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:31.450443 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141cf213-3483-4d82-b0a6-3b3126e39c1d None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:31.450729 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141cf213-3483-4d82-b0a6-3b3126e39c1d None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:31.451005 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141cf213-3483-4d82-b0a6-3b3126e39c1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:31.520843 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-141cf213-3483-4d82-b0a6-3b3126e39c1d None None] MFA Rules not processed for user `28da55f3d8da4bc486c210bb198249f4`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:31.522173 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-141cf213-3483-4d82-b0a6-3b3126e39c1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:31.569195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84649c9-a32e-4100-bfa3-d89600da1b74 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:31.571529 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84649c9-a32e-4100-bfa3-d89600da1b74 service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:31.574346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:31.587311 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 942/1916] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:31 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:31.589010 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44eb177e-2700-4d10-8e84-90bc31e616f7 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:31.589406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44eb177e-2700-4d10-8e84-90bc31e616f7 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:31.589785 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44eb177e-2700-4d10-8e84-90bc31e616f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:31.629312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-44eb177e-2700-4d10-8e84-90bc31e616f7 None None] MFA Rules not processed for user `2b50e2d42f2541f1af0dd4f4e55fe7c1`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:31.630129 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-44eb177e-2700-4d10-8e84-90bc31e616f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:31.663349 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 975/1917] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:31 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:31.668095 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-141cf213-3483-4d82-b0a6-3b3126e39c1d None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:31.751120 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 943/1918] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:31 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:31.765767 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44eb177e-2700-4d10-8e84-90bc31e616f7 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:31.767945 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-44eb177e-2700-4d10-8e84-90bc31e616f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:31.775727 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-141cf213-3483-4d82-b0a6-3b3126e39c1d None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:31.788340 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e15cf1-6c79-40ac-a507-d5d057685e98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:31.789715 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e15cf1-6c79-40ac-a507-d5d057685e98 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:31.790071 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e15cf1-6c79-40ac-a507-d5d057685e98 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:31.790355 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e15cf1-6c79-40ac-a507-d5d057685e98 admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:31.791017 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e15cf1-6c79-40ac-a507-d5d057685e98 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:31.798404 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e15cf1-6c79-40ac-a507-d5d057685e98 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:31.854018 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44eb177e-2700-4d10-8e84-90bc31e616f7 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:31.859992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45f9085-bc06-4319-ad3c-556050193760 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:31.860778 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45f9085-bc06-4319-ad3c-556050193760 service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:31.861283 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45f9085-bc06-4319-ad3c-556050193760 service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:31.861754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45f9085-bc06-4319-ad3c-556050193760 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:31.873635 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 976/1919] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:47:31 2022] POST /identity/v3/projects => generated 358 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 15:47:31.877699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e15cf1-6c79-40ac-a507-d5d057685e98 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:31.881227 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45f9085-bc06-4319-ad3c-556050193760 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:31.883227 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45f9085-bc06-4319-ad3c-556050193760 service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:31.915428 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 944/1920] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:31 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:31.917835 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916773ec-8be7-4662-8f58-669de7516e52 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:31.918131 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916773ec-8be7-4662-8f58-669de7516e52 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:31.918406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916773ec-8be7-4662-8f58-669de7516e52 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:31.922737 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 945/1921] 10.222.0.102 () {54 vars in 926 bytes} [Wed Aug 24 15:47:31 2022] GET /identity => generated 271 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:47:31.954747 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965c7ec8-3d11-4528-8b5d-7c9ab02a5e2b None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:31.955037 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965c7ec8-3d11-4528-8b5d-7c9ab02a5e2b None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:31.955338 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965c7ec8-3d11-4528-8b5d-7c9ab02a5e2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:31.983341 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-965c7ec8-3d11-4528-8b5d-7c9ab02a5e2b None None] MFA Rules not processed for user `e0d544379fe44438ad10289b45124374`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:31.984640 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-965c7ec8-3d11-4528-8b5d-7c9ab02a5e2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:32.030929 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:32.070529 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e15cf1-6c79-40ac-a507-d5d057685e98 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:32.077553 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 946/1922] 10.222.0.102 () {58 vars in 1023 bytes} [Wed Aug 24 15:47:31 2022] POST /identity/v3/auth/tokens => generated 3382 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:47:32.091208 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f1925b-12bf-4397-96f2-e672f67dac7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:32.091908 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f1925b-12bf-4397-96f2-e672f67dac7e admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:32.092201 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f1925b-12bf-4397-96f2-e672f67dac7e admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:32.103655 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f1925b-12bf-4397-96f2-e672f67dac7e admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:32.105446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f1925b-12bf-4397-96f2-e672f67dac7e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:32.114334 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f1925b-12bf-4397-96f2-e672f67dac7e admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:32.160537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 977/1923] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:47:31 2022] POST /identity/v3/users => generated 286 bytes in 283 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:32.166323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965c7ec8-3d11-4528-8b5d-7c9ab02a5e2b None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:32.183262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965c7ec8-3d11-4528-8b5d-7c9ab02a5e2b None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:32.188763 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a3bfee-52e5-4c9c-bc22-f41a280c2d14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:32.189566 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a3bfee-52e5-4c9c-bc22-f41a280c2d14 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:32.189856 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a3bfee-52e5-4c9c-bc22-f41a280c2d14 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:32.190124 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a3bfee-52e5-4c9c-bc22-f41a280c2d14 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:32.190787 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a3bfee-52e5-4c9c-bc22-f41a280c2d14 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:32.193376 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a3bfee-52e5-4c9c-bc22-f41a280c2d14 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:32.215517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 947/1924] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:32 2022] GET /identity/v3/roles => generated 1427 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:32.218165 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f1925b-12bf-4397-96f2-e672f67dac7e admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:32.238567 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f1925b-12bf-4397-96f2-e672f67dac7e admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:32.251198 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4228b01-a05d-44b2-86a4-36b4c36fe7e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:32.254125 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4228b01-a05d-44b2-86a4-36b4c36fe7e2 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:32.254492 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4228b01-a05d-44b2-86a4-36b4c36fe7e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:32.254780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4228b01-a05d-44b2-86a4-36b4c36fe7e2 admin admin] PATH_INFO: `/v3/projects/39d3d22eb8a945abb61dd8a082a3b57f/users/ddde15b3f78242a689a9a05b82fbc9d0/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:32.276155 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4228b01-a05d-44b2-86a4-36b4c36fe7e2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=39d3d22eb8a945abb61dd8a082a3b57f, user_id=ddde15b3f78242a689a9a05b82fbc9d0, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:32.283719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4228b01-a05d-44b2-86a4-36b4c36fe7e2 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:32.301236 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 978/1925] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:47:32 2022] PUT /identity/v3/projects/39d3d22eb8a945abb61dd8a082a3b57f/users/ddde15b3f78242a689a9a05b82fbc9d0/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:32.306791 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a3bfee-52e5-4c9c-bc22-f41a280c2d14 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:32.322935 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a3bfee-52e5-4c9c-bc22-f41a280c2d14 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:32.475866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60db697-925d-4251-a945-0007cc9d120c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:32.477361 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60db697-925d-4251-a945-0007cc9d120c admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:32.478456 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60db697-925d-4251-a945-0007cc9d120c admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:32.480931 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60db697-925d-4251-a945-0007cc9d120c admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:32.482109 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e60db697-925d-4251-a945-0007cc9d120c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:32.486666 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e60db697-925d-4251-a945-0007cc9d120c admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:32.519140 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4228b01-a05d-44b2-86a4-36b4c36fe7e2 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:32.532373 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 948/1926] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:32 2022] GET /identity/v3/roles => generated 1427 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:32.536062 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4228b01-a05d-44b2-86a4-36b4c36fe7e2 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:32.548615 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8450ff60-1ac5-4b92-8d9f-82f7b666c603 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:32.549336 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8450ff60-1ac5-4b92-8d9f-82f7b666c603 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:32.549621 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8450ff60-1ac5-4b92-8d9f-82f7b666c603 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:32.549891 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8450ff60-1ac5-4b92-8d9f-82f7b666c603 admin admin] PATH_INFO: `/v3/projects/39d3d22eb8a945abb61dd8a082a3b57f/users/ddde15b3f78242a689a9a05b82fbc9d0/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:32.559680 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8450ff60-1ac5-4b92-8d9f-82f7b666c603 admin admin] RBAC: Authorizing `identity:create_grant(project_id=39d3d22eb8a945abb61dd8a082a3b57f, user_id=ddde15b3f78242a689a9a05b82fbc9d0, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:32.562349 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8450ff60-1ac5-4b92-8d9f-82f7b666c603 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:32.593443 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 979/1927] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:47:32 2022] PUT /identity/v3/projects/39d3d22eb8a945abb61dd8a082a3b57f/users/ddde15b3f78242a689a9a05b82fbc9d0/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:32.593681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60db697-925d-4251-a945-0007cc9d120c admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:32.606732 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-e60db697-925d-4251-a945-0007cc9d120c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:32.629997 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0702697e-741c-4170-9da8-26600f87d01c None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:32.630293 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0702697e-741c-4170-9da8-26600f87d01c None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:32.634810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0702697e-741c-4170-9da8-26600f87d01c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:32.692521 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:32.708654 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-0702697e-741c-4170-9da8-26600f87d01c None None] MFA Rules not processed for user `ddde15b3f78242a689a9a05b82fbc9d0`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:32.709919 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-0702697e-741c-4170-9da8-26600f87d01c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:32.772702 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60db697-925d-4251-a945-0007cc9d120c admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:32.850118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:32.850287 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 980/1928] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:32 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:32.861053 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0702697e-741c-4170-9da8-26600f87d01c None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:32.871778 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d00fa9b-b266-4520-ad04-fcce3aef64c6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:32.872590 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d00fa9b-b266-4520-ad04-fcce3aef64c6 service nova] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:32.872876 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d00fa9b-b266-4520-ad04-fcce3aef64c6 service nova] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:32.873152 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d00fa9b-b266-4520-ad04-fcce3aef64c6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:32.884089 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d00fa9b-b266-4520-ad04-fcce3aef64c6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:32.885700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d00fa9b-b266-4520-ad04-fcce3aef64c6 service nova] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:32.899506 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 949/1929] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:32 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:32.946495 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0702697e-741c-4170-9da8-26600f87d01c None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:32.950773 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47dd282-7520-4c04-8906-873c7564d885 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:32.951417 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47dd282-7520-4c04-8906-873c7564d885 service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:32.951789 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47dd282-7520-4c04-8906-873c7564d885 service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:32.952058 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47dd282-7520-4c04-8906-873c7564d885 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:32.971414 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47dd282-7520-4c04-8906-873c7564d885 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:32.973246 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47dd282-7520-4c04-8906-873c7564d885 service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:32.985073 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 981/1930] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:32 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 15:47:33.151237 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d00fa9b-b266-4520-ad04-fcce3aef64c6 service nova] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:33.169038 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d00fa9b-b266-4520-ad04-fcce3aef64c6 service nova] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:33.174846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1c85a5-1f33-45f3-b5cc-0807162e7a6c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:33.176035 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1c85a5-1f33-45f3-b5cc-0807162e7a6c service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:33.176206 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1c85a5-1f33-45f3-b5cc-0807162e7a6c service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:33.176447 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1c85a5-1f33-45f3-b5cc-0807162e7a6c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:33.310069 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f1c85a5-1f33-45f3-b5cc-0807162e7a6c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:33.311925 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f1c85a5-1f33-45f3-b5cc-0807162e7a6c service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:33.325208 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 950/1931] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:33 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:38.057395 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47dd282-7520-4c04-8906-873c7564d885 service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:38.065893 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47dd282-7520-4c04-8906-873c7564d885 service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:38.069955 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9522e5e2-e799-49ca-8355-fddcec3f04ed service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:38.071128 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9522e5e2-e799-49ca-8355-fddcec3f04ed service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:38.071410 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9522e5e2-e799-49ca-8355-fddcec3f04ed service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:38.071726 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9522e5e2-e799-49ca-8355-fddcec3f04ed service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:38.144979 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9522e5e2-e799-49ca-8355-fddcec3f04ed service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:38.146450 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9522e5e2-e799-49ca-8355-fddcec3f04ed service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:38.158079 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 982/1932] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:38 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:43.319247 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=38 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:44.965280 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1c85a5-1f33-45f3-b5cc-0807162e7a6c service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:44.966902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-6f1c85a5-1f33-45f3-b5cc-0807162e7a6c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:45.033067 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1c85a5-1f33-45f3-b5cc-0807162e7a6c service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:45.037407 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b85e3a-4000-4656-9424-376de4333dbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:45.038028 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b85e3a-4000-4656-9424-376de4333dbe admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:45.038303 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b85e3a-4000-4656-9424-376de4333dbe admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:45.038578 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b85e3a-4000-4656-9424-376de4333dbe admin admin] PATH_INFO: `/v3/users/a39c02c655db4db08180a82f2082798b` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:45.039810 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9b85e3a-4000-4656-9424-376de4333dbe admin admin] RBAC: Authorizing `identity:delete_user(user_id=a39c02c655db4db08180a82f2082798b)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:45.041714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9b85e3a-4000-4656-9424-376de4333dbe admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:45.093471 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-b9b85e3a-4000-4656-9424-376de4333dbe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a39c02c655db4db08180a82f2082798b'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:45.094183 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-b9b85e3a-4000-4656-9424-376de4333dbe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a39c02c655db4db08180a82f2082798b'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:45.101463 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-b9b85e3a-4000-4656-9424-376de4333dbe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a39c02c655db4db08180a82f2082798b'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:45.113835 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-b9b85e3a-4000-4656-9424-376de4333dbe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a39c02c655db4db08180a82f2082798b'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:45.114533 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-b9b85e3a-4000-4656-9424-376de4333dbe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a39c02c655db4db08180a82f2082798b'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:45.129026 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 951/1933] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:47:44 2022] DELETE /identity/v3/users/a39c02c655db4db08180a82f2082798b => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:45.135789 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a906c38-e155-4e65-9f27-e56338fbb15d None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:45.135789 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a906c38-e155-4e65-9f27-e56338fbb15d None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:45.135789 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a906c38-e155-4e65-9f27-e56338fbb15d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:45.157446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-4a906c38-e155-4e65-9f27-e56338fbb15d None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:45.158652 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-4a906c38-e155-4e65-9f27-e56338fbb15d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:45.186259 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:45.222301 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 983/1934] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:45 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:45.233077 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b85e3a-4000-4656-9424-376de4333dbe admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:45.234482 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-b9b85e3a-4000-4656-9424-376de4333dbe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:45.315381 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b85e3a-4000-4656-9424-376de4333dbe admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:45.321825 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e29d0a-d045-4a6b-b63c-93f69db55919 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:45.322843 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e29d0a-d045-4a6b-b63c-93f69db55919 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:45.323322 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e29d0a-d045-4a6b-b63c-93f69db55919 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:45.323903 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e29d0a-d045-4a6b-b63c-93f69db55919 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:45.395962 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e29d0a-d045-4a6b-b63c-93f69db55919 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:45.397448 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e29d0a-d045-4a6b-b63c-93f69db55919 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:45.418842 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a906c38-e155-4e65-9f27-e56338fbb15d None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:45.420848 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-4a906c38-e155-4e65-9f27-e56338fbb15d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:45.427193 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 952/1935] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:45 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:45.443168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a906c38-e155-4e65-9f27-e56338fbb15d None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:45.454945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7efccf7f-0b10-4cb6-b971-9caa6d7dc76b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:45.455897 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efccf7f-0b10-4cb6-b971-9caa6d7dc76b admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:45.456345 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efccf7f-0b10-4cb6-b971-9caa6d7dc76b admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:45.456791 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efccf7f-0b10-4cb6-b971-9caa6d7dc76b admin admin] PATH_INFO: `/v3/users/2b50e2d42f2541f1af0dd4f4e55fe7c1` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:45.460278 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7efccf7f-0b10-4cb6-b971-9caa6d7dc76b admin admin] RBAC: Authorizing `identity:delete_user(user_id=2b50e2d42f2541f1af0dd4f4e55fe7c1)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:45.462899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7efccf7f-0b10-4cb6-b971-9caa6d7dc76b admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:45.602740 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-7efccf7f-0b10-4cb6-b971-9caa6d7dc76b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2b50e2d42f2541f1af0dd4f4e55fe7c1'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:45.603558 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-7efccf7f-0b10-4cb6-b971-9caa6d7dc76b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2b50e2d42f2541f1af0dd4f4e55fe7c1'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:45.605602 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-7efccf7f-0b10-4cb6-b971-9caa6d7dc76b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2b50e2d42f2541f1af0dd4f4e55fe7c1'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:45.624281 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-7efccf7f-0b10-4cb6-b971-9caa6d7dc76b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2b50e2d42f2541f1af0dd4f4e55fe7c1'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:45.649579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-7efccf7f-0b10-4cb6-b971-9caa6d7dc76b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2b50e2d42f2541f1af0dd4f4e55fe7c1'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:45.671471 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 984/1936] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:47:45 2022] DELETE /identity/v3/users/2b50e2d42f2541f1af0dd4f4e55fe7c1 => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:45.675936 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3005cbcb-f9fa-472a-a993-956916966d4c None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:45.676562 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3005cbcb-f9fa-472a-a993-956916966d4c None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:45.676860 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3005cbcb-f9fa-472a-a993-956916966d4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:45.701838 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-3005cbcb-f9fa-472a-a993-956916966d4c None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:45.703497 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-3005cbcb-f9fa-472a-a993-956916966d4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:45.733666 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:45.826861 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 953/1937] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:45 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:45.837385 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7efccf7f-0b10-4cb6-b971-9caa6d7dc76b admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:45.839294 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-7efccf7f-0b10-4cb6-b971-9caa6d7dc76b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:45.961594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7efccf7f-0b10-4cb6-b971-9caa6d7dc76b admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:45.967409 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e3137f-ded5-4470-a587-12bba24f49a6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:45.968193 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e3137f-ded5-4470-a587-12bba24f49a6 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:45.968193 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e3137f-ded5-4470-a587-12bba24f49a6 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:45.968635 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e3137f-ded5-4470-a587-12bba24f49a6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:45.982695 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e3137f-ded5-4470-a587-12bba24f49a6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:45.984356 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e3137f-ded5-4470-a587-12bba24f49a6 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:46.015198 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 985/1938] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:45 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:46.065556 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3005cbcb-f9fa-472a-a993-956916966d4c None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:46.068321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-3005cbcb-f9fa-472a-a993-956916966d4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:46.083459 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3005cbcb-f9fa-472a-a993-956916966d4c None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:46.089083 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa431890-3797-4a30-aa7f-c4ab0cc3c12a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:46.089725 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa431890-3797-4a30-aa7f-c4ab0cc3c12a admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:46.090082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa431890-3797-4a30-aa7f-c4ab0cc3c12a admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:46.090271 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa431890-3797-4a30-aa7f-c4ab0cc3c12a admin admin] PATH_INFO: `/v3/projects/e57aad5c0f1147a3a95508e38d10cf78` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:46.091510 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa431890-3797-4a30-aa7f-c4ab0cc3c12a admin admin] RBAC: Authorizing `identity:delete_project(project_id=e57aad5c0f1147a3a95508e38d10cf78)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:46.093596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa431890-3797-4a30-aa7f-c4ab0cc3c12a admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:46.159587 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-fa431890-3797-4a30-aa7f-c4ab0cc3c12a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e57aad5c0f1147a3a95508e38d10cf78', 'request_id': 'req-fa431890-3797-4a30-aa7f-c4ab0cc3c12a'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:46.167943 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-fa431890-3797-4a30-aa7f-c4ab0cc3c12a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e57aad5c0f1147a3a95508e38d10cf78', 'request_id': 'req-fa431890-3797-4a30-aa7f-c4ab0cc3c12a'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:46.177477 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-fa431890-3797-4a30-aa7f-c4ab0cc3c12a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e57aad5c0f1147a3a95508e38d10cf78', 'request_id': 'req-fa431890-3797-4a30-aa7f-c4ab0cc3c12a'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:46.194808 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-fa431890-3797-4a30-aa7f-c4ab0cc3c12a admin admin] The token cache is being invalidate because project e57aad5c0f1147a3a95508e38d10cf78 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:47:46.195961 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-fa431890-3797-4a30-aa7f-c4ab0cc3c12a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:46.196892 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-fa431890-3797-4a30-aa7f-c4ab0cc3c12a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:46.198599 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 954/1939] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:47:46 2022] DELETE /identity/v3/projects/e57aad5c0f1147a3a95508e38d10cf78 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:46.202899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e3137f-ded5-4470-a587-12bba24f49a6 service neutron] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:46.214149 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e3137f-ded5-4470-a587-12bba24f49a6 service neutron] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:46.287575 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:46.303243 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007dd407-5e3b-4aa9-8312-14ebc8139228 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:46.305139 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007dd407-5e3b-4aa9-8312-14ebc8139228 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:46.305139 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007dd407-5e3b-4aa9-8312-14ebc8139228 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:46.305139 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007dd407-5e3b-4aa9-8312-14ebc8139228 admin admin] PATH_INFO: `/v3/users/3931c15eb05e4a3185e7a4dee7d21a52` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:46.306028 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007dd407-5e3b-4aa9-8312-14ebc8139228 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3931c15eb05e4a3185e7a4dee7d21a52)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:46.308183 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007dd407-5e3b-4aa9-8312-14ebc8139228 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:46.381366 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-007dd407-5e3b-4aa9-8312-14ebc8139228 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3931c15eb05e4a3185e7a4dee7d21a52'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:46.383402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-007dd407-5e3b-4aa9-8312-14ebc8139228 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3931c15eb05e4a3185e7a4dee7d21a52'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:46.383402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-007dd407-5e3b-4aa9-8312-14ebc8139228 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3931c15eb05e4a3185e7a4dee7d21a52'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:46.393754 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-007dd407-5e3b-4aa9-8312-14ebc8139228 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3931c15eb05e4a3185e7a4dee7d21a52'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:46.430331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa431890-3797-4a30-aa7f-c4ab0cc3c12a admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:46.432710 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-fa431890-3797-4a30-aa7f-c4ab0cc3c12a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:46.433116 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-007dd407-5e3b-4aa9-8312-14ebc8139228 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3931c15eb05e4a3185e7a4dee7d21a52'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:46.450165 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa431890-3797-4a30-aa7f-c4ab0cc3c12a admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:46.453021 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 986/1940] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:47:46 2022] DELETE /identity/v3/users/3931c15eb05e4a3185e7a4dee7d21a52 => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:46.547936 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db49977-319b-4127-8bb1-f2755d777ef3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:46.548967 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db49977-319b-4127-8bb1-f2755d777ef3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:46.548967 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db49977-319b-4127-8bb1-f2755d777ef3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:46.549142 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db49977-319b-4127-8bb1-f2755d777ef3 admin admin] PATH_INFO: `/v3/projects/cb0b0652e3df461093681ec09ffbf121` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:46.550308 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db49977-319b-4127-8bb1-f2755d777ef3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cb0b0652e3df461093681ec09ffbf121)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:46.552565 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db49977-319b-4127-8bb1-f2755d777ef3 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:46.609224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:46.611207 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-2db49977-319b-4127-8bb1-f2755d777ef3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cb0b0652e3df461093681ec09ffbf121', 'request_id': 'req-2db49977-319b-4127-8bb1-f2755d777ef3'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:46.620939 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-2db49977-319b-4127-8bb1-f2755d777ef3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cb0b0652e3df461093681ec09ffbf121', 'request_id': 'req-2db49977-319b-4127-8bb1-f2755d777ef3'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:46.634888 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-2db49977-319b-4127-8bb1-f2755d777ef3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cb0b0652e3df461093681ec09ffbf121', 'request_id': 'req-2db49977-319b-4127-8bb1-f2755d777ef3'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:46.655318 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-2db49977-319b-4127-8bb1-f2755d777ef3 admin admin] The token cache is being invalidate because project cb0b0652e3df461093681ec09ffbf121 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:47:46.655703 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-2db49977-319b-4127-8bb1-f2755d777ef3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:46.656370 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-2db49977-319b-4127-8bb1-f2755d777ef3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:46.657943 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 955/1941] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:47:46 2022] DELETE /identity/v3/projects/cb0b0652e3df461093681ec09ffbf121 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 15:47:46.663195 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4199bf66-f4dc-4d92-864c-e25d57fbd1dd None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:46.663489 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4199bf66-f4dc-4d92-864c-e25d57fbd1dd None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:46.663799 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4199bf66-f4dc-4d92-864c-e25d57fbd1dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:46.688651 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-4199bf66-f4dc-4d92-864c-e25d57fbd1dd None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:46.689901 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-4199bf66-f4dc-4d92-864c-e25d57fbd1dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:46.755142 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 987/1942] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:46 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:46.764694 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07778326-9477-4b6f-a993-91d11ef112f7 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:46.764895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07778326-9477-4b6f-a993-91d11ef112f7 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:46.765119 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07778326-9477-4b6f-a993-91d11ef112f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:46.821066 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-07778326-9477-4b6f-a993-91d11ef112f7 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:46.827438 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-07778326-9477-4b6f-a993-91d11ef112f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:46.848010 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 956/1943] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:46 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:46.848408 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4199bf66-f4dc-4d92-864c-e25d57fbd1dd None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:46.869316 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4199bf66-f4dc-4d92-864c-e25d57fbd1dd None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:46.874664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83998a66-bda4-47af-83f6-10aac00ae065 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:46.875353 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83998a66-bda4-47af-83f6-10aac00ae065 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:46.875664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83998a66-bda4-47af-83f6-10aac00ae065 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:46.875977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83998a66-bda4-47af-83f6-10aac00ae065 admin admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:46.876637 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83998a66-bda4-47af-83f6-10aac00ae065 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:47:46.876941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83998a66-bda4-47af-83f6-10aac00ae065 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:46.878534 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83998a66-bda4-47af-83f6-10aac00ae065 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:46.894719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 988/1944] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:47:46 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:47:46.896014 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07778326-9477-4b6f-a993-91d11ef112f7 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:46.906246 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07778326-9477-4b6f-a993-91d11ef112f7 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:46.911186 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbb3d65-9953-4713-b333-c898c4f8c95c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:46.911844 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbb3d65-9953-4713-b333-c898c4f8c95c admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:46.912129 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbb3d65-9953-4713-b333-c898c4f8c95c admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:46.912406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbb3d65-9953-4713-b333-c898c4f8c95c admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:46.913058 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbb3d65-9953-4713-b333-c898c4f8c95c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:46.915874 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbb3d65-9953-4713-b333-c898c4f8c95c admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:46.951369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 957/1945] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:47:46 2022] POST /identity/v3/projects => generated 376 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:46.952757 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83998a66-bda4-47af-83f6-10aac00ae065 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:46.964008 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83998a66-bda4-47af-83f6-10aac00ae065 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:47.038538 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:47.051054 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35dd456-8393-4ddc-ae22-a05579cca754 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:47.051858 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35dd456-8393-4ddc-ae22-a05579cca754 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:47.052229 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35dd456-8393-4ddc-ae22-a05579cca754 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:47.052602 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35dd456-8393-4ddc-ae22-a05579cca754 admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:47.053360 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d35dd456-8393-4ddc-ae22-a05579cca754 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:47.056059 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d35dd456-8393-4ddc-ae22-a05579cca754 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:47.080670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 989/1946] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:47:46 2022] POST /identity/v3/users => generated 289 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 15:47:47.085847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbb3d65-9953-4713-b333-c898c4f8c95c admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:47.098644 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbb3d65-9953-4713-b333-c898c4f8c95c admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:47.105284 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24a0d8b-39cb-4036-90e0-551944561be2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:47.105896 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24a0d8b-39cb-4036-90e0-551944561be2 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:47.106179 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24a0d8b-39cb-4036-90e0-551944561be2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:47.106452 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24a0d8b-39cb-4036-90e0-551944561be2 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:47.107106 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24a0d8b-39cb-4036-90e0-551944561be2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:47.108752 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24a0d8b-39cb-4036-90e0-551944561be2 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:47.122884 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 958/1947] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:47 2022] GET /identity/v3/roles => generated 1427 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:47.124345 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35dd456-8393-4ddc-ae22-a05579cca754 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:47.137347 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35dd456-8393-4ddc-ae22-a05579cca754 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:47.144263 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7db1f6-eb17-422e-87de-d37930066fb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:47.144876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7db1f6-eb17-422e-87de-d37930066fb7 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:47.145157 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7db1f6-eb17-422e-87de-d37930066fb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:47.145444 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7db1f6-eb17-422e-87de-d37930066fb7 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:47.146100 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7db1f6-eb17-422e-87de-d37930066fb7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:47.147908 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7db1f6-eb17-422e-87de-d37930066fb7 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:47.179271 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 990/1948] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:47 2022] GET /identity/v3/roles => generated 1427 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:47.182662 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24a0d8b-39cb-4036-90e0-551944561be2 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:47.187996 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7db1f6-eb17-422e-87de-d37930066fb7 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:47.189757 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-0a7db1f6-eb17-422e-87de-d37930066fb7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:47.196133 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24a0d8b-39cb-4036-90e0-551944561be2 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:47.202139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89c1d1a-6037-49ed-b66d-2e85b8acc7e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:47.202811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89c1d1a-6037-49ed-b66d-2e85b8acc7e8 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:47.203089 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89c1d1a-6037-49ed-b66d-2e85b8acc7e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:47.203364 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89c1d1a-6037-49ed-b66d-2e85b8acc7e8 admin admin] PATH_INFO: `/v3/projects/4153c9ca4b2d4c6caf990f13aa45352a/users/eb357d729fa2427fab76907920e8fd2e/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:47.204620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7db1f6-eb17-422e-87de-d37930066fb7 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:47.212643 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c953ec8-52ae-4ebc-b0b4-de8ceb891139 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:47.214034 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c953ec8-52ae-4ebc-b0b4-de8ceb891139 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:47.214515 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c953ec8-52ae-4ebc-b0b4-de8ceb891139 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:47.216034 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c953ec8-52ae-4ebc-b0b4-de8ceb891139 admin admin] PATH_INFO: `/v3/projects/f0a91faefeb54ee5a64cadeed8d8a524` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:47.220055 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c953ec8-52ae-4ebc-b0b4-de8ceb891139 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f0a91faefeb54ee5a64cadeed8d8a524)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:47.222010 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c953ec8-52ae-4ebc-b0b4-de8ceb891139 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:47.240432 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c89c1d1a-6037-49ed-b66d-2e85b8acc7e8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4153c9ca4b2d4c6caf990f13aa45352a, user_id=eb357d729fa2427fab76907920e8fd2e, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:47.244165 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c89c1d1a-6037-49ed-b66d-2e85b8acc7e8 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:47.260274 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 959/1949] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:47:47 2022] PUT /identity/v3/projects/4153c9ca4b2d4c6caf990f13aa45352a/users/eb357d729fa2427fab76907920e8fd2e/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:47.270603 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b8d480-4337-48c7-9ae1-c1d078242303 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:47.270961 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b8d480-4337-48c7-9ae1-c1d078242303 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:47.271312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b8d480-4337-48c7-9ae1-c1d078242303 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:47.275942 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-6c953ec8-52ae-4ebc-b0b4-de8ceb891139 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f0a91faefeb54ee5a64cadeed8d8a524', 'request_id': 'req-6c953ec8-52ae-4ebc-b0b4-de8ceb891139'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:47.282537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-6c953ec8-52ae-4ebc-b0b4-de8ceb891139 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f0a91faefeb54ee5a64cadeed8d8a524', 'request_id': 'req-6c953ec8-52ae-4ebc-b0b4-de8ceb891139'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:47.289499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-6c953ec8-52ae-4ebc-b0b4-de8ceb891139 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f0a91faefeb54ee5a64cadeed8d8a524', 'request_id': 'req-6c953ec8-52ae-4ebc-b0b4-de8ceb891139'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:47.301695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-09b8d480-4337-48c7-9ae1-c1d078242303 None None] MFA Rules not processed for user `eb357d729fa2427fab76907920e8fd2e`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:47.303000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-09b8d480-4337-48c7-9ae1-c1d078242303 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:47.304170 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-6c953ec8-52ae-4ebc-b0b4-de8ceb891139 admin admin] The token cache is being invalidate because project f0a91faefeb54ee5a64cadeed8d8a524 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:47:47.304497 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-6c953ec8-52ae-4ebc-b0b4-de8ceb891139 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:47.305371 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-6c953ec8-52ae-4ebc-b0b4-de8ceb891139 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:47.307677 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 991/1950] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:47:47 2022] DELETE /identity/v3/projects/f0a91faefeb54ee5a64cadeed8d8a524 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 15:47:47.312743 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14ab6c1-41fb-4a51-b7f8-5c7b5e981528 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:47.313043 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14ab6c1-41fb-4a51-b7f8-5c7b5e981528 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:47.313325 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14ab6c1-41fb-4a51-b7f8-5c7b5e981528 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:47.337488 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-f14ab6c1-41fb-4a51-b7f8-5c7b5e981528 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:47.338692 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-f14ab6c1-41fb-4a51-b7f8-5c7b5e981528 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:47.375791 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 960/1951] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:47 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:47:47.386566 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b8d480-4337-48c7-9ae1-c1d078242303 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:47.388221 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-09b8d480-4337-48c7-9ae1-c1d078242303 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:47.403675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 992/1952] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:47 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:47:47.411131 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3446dc31-55dc-416b-b6dc-baf11b258320 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:47.411425 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3446dc31-55dc-416b-b6dc-baf11b258320 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:47.411740 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3446dc31-55dc-416b-b6dc-baf11b258320 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:47.432292 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-3446dc31-55dc-416b-b6dc-baf11b258320 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:47.433482 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-3446dc31-55dc-416b-b6dc-baf11b258320 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:47.441839 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 993/1953] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:47 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:47:47.446076 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3446dc31-55dc-416b-b6dc-baf11b258320 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:47.450824 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b8d480-4337-48c7-9ae1-c1d078242303 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:47.456273 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8a4b6c-0b1b-43c4-a20e-28da91472aaf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:47.456738 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3446dc31-55dc-416b-b6dc-baf11b258320 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:47.456916 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8a4b6c-0b1b-43c4-a20e-28da91472aaf service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:47.457360 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8a4b6c-0b1b-43c4-a20e-28da91472aaf service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:47.457643 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8a4b6c-0b1b-43c4-a20e-28da91472aaf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:47.462495 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7abd416-5c53-4f19-9d95-8b90bfb251d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:47.463122 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7abd416-5c53-4f19-9d95-8b90bfb251d0 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:47.463404 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7abd416-5c53-4f19-9d95-8b90bfb251d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:47.463698 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7abd416-5c53-4f19-9d95-8b90bfb251d0 admin admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:47.464385 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7abd416-5c53-4f19-9d95-8b90bfb251d0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:47:47.464670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7abd416-5c53-4f19-9d95-8b90bfb251d0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:47.466395 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7abd416-5c53-4f19-9d95-8b90bfb251d0 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:47.468875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f8a4b6c-0b1b-43c4-a20e-28da91472aaf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:47.470395 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f8a4b6c-0b1b-43c4-a20e-28da91472aaf service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:47.482353 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 961/1954] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:47 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 15:47:47.484766 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 994/1955] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:47:47 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:47:47.488619 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8a4b6c-0b1b-43c4-a20e-28da91472aaf service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:47.500663 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=94,keystone:DELETE=3,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:47.505340 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8a4b6c-0b1b-43c4-a20e-28da91472aaf service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:47.517894 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895f0e1d-95a8-42cc-bf26-db01bcfa3c7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:47.517894 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895f0e1d-95a8-42cc-bf26-db01bcfa3c7f admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:47.517894 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895f0e1d-95a8-42cc-bf26-db01bcfa3c7f admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:47.517894 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895f0e1d-95a8-42cc-bf26-db01bcfa3c7f admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:47.518359 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895f0e1d-95a8-42cc-bf26-db01bcfa3c7f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:47.518359 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895f0e1d-95a8-42cc-bf26-db01bcfa3c7f admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:47.546816 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 962/1956] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:47:47 2022] POST /identity/v3/projects => generated 374 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 15:47:47.551168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7abd416-5c53-4f19-9d95-8b90bfb251d0 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:47.561540 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7abd416-5c53-4f19-9d95-8b90bfb251d0 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:47.597149 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:47.631312 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04d06ae-6b95-427e-8c57-8e536e2c2ec6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:47.632092 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04d06ae-6b95-427e-8c57-8e536e2c2ec6 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:47.632995 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04d06ae-6b95-427e-8c57-8e536e2c2ec6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:47.633473 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04d06ae-6b95-427e-8c57-8e536e2c2ec6 admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:47.634360 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04d06ae-6b95-427e-8c57-8e536e2c2ec6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:47.637053 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04d06ae-6b95-427e-8c57-8e536e2c2ec6 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:47.666185 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 995/1957] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:47:47 2022] POST /identity/v3/users => generated 288 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:47.667398 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895f0e1d-95a8-42cc-bf26-db01bcfa3c7f admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:47.677994 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895f0e1d-95a8-42cc-bf26-db01bcfa3c7f admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:47.683188 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1281e3a-fd7b-4632-89ab-618830673fe5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:47.683960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1281e3a-fd7b-4632-89ab-618830673fe5 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:47.684088 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1281e3a-fd7b-4632-89ab-618830673fe5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:47.684359 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1281e3a-fd7b-4632-89ab-618830673fe5 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:47.685055 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1281e3a-fd7b-4632-89ab-618830673fe5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:47.686665 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1281e3a-fd7b-4632-89ab-618830673fe5 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:47.700932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 963/1958] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:47 2022] GET /identity/v3/roles => generated 1427 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:47.705086 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04d06ae-6b95-427e-8c57-8e536e2c2ec6 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:47.717692 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04d06ae-6b95-427e-8c57-8e536e2c2ec6 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:47.722366 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faaf04d5-3f9b-4521-8124-a4bd80edd533 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:47.722967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faaf04d5-3f9b-4521-8124-a4bd80edd533 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:47.723256 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faaf04d5-3f9b-4521-8124-a4bd80edd533 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:47.723530 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faaf04d5-3f9b-4521-8124-a4bd80edd533 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:47.724240 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faaf04d5-3f9b-4521-8124-a4bd80edd533 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:47.726103 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faaf04d5-3f9b-4521-8124-a4bd80edd533 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:47.739658 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 996/1959] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:47 2022] GET /identity/v3/roles => generated 1427 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:47.743776 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1281e3a-fd7b-4632-89ab-618830673fe5 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:47.764846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1281e3a-fd7b-4632-89ab-618830673fe5 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:47.770615 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c66577-12c5-4e1c-93ee-a31f1ce2eb95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:47.771376 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c66577-12c5-4e1c-93ee-a31f1ce2eb95 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:47.771874 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c66577-12c5-4e1c-93ee-a31f1ce2eb95 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:47.772256 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c66577-12c5-4e1c-93ee-a31f1ce2eb95 admin admin] PATH_INFO: `/v3/projects/26e324399ad64863a994246cc04a4751/users/1e34111a3b2f4c1fb75252c4806d3da4/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:47.792416 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11c66577-12c5-4e1c-93ee-a31f1ce2eb95 admin admin] RBAC: Authorizing `identity:create_grant(project_id=26e324399ad64863a994246cc04a4751, user_id=1e34111a3b2f4c1fb75252c4806d3da4, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:47.795130 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11c66577-12c5-4e1c-93ee-a31f1ce2eb95 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:47.819259 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 964/1960] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:47:47 2022] PUT /identity/v3/projects/26e324399ad64863a994246cc04a4751/users/1e34111a3b2f4c1fb75252c4806d3da4/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 15:47:47.825024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56119430-e499-4eec-a025-c1f8a853a70e None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:47.825578 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56119430-e499-4eec-a025-c1f8a853a70e None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:47.826719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56119430-e499-4eec-a025-c1f8a853a70e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:47.853314 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-56119430-e499-4eec-a025-c1f8a853a70e None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:47.854625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-56119430-e499-4eec-a025-c1f8a853a70e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:47.925667 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 997/1961] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:47 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:47.936155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c66577-12c5-4e1c-93ee-a31f1ce2eb95 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:47.937951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-11c66577-12c5-4e1c-93ee-a31f1ce2eb95 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:47.998957 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c66577-12c5-4e1c-93ee-a31f1ce2eb95 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:48.003474 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fe57cf-3fbe-46a5-a4b3-f85d89cabc58 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:48.004291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fe57cf-3fbe-46a5-a4b3-f85d89cabc58 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:48.004724 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fe57cf-3fbe-46a5-a4b3-f85d89cabc58 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:48.005172 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fe57cf-3fbe-46a5-a4b3-f85d89cabc58 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:48.015920 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87fe57cf-3fbe-46a5-a4b3-f85d89cabc58 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:48.017784 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87fe57cf-3fbe-46a5-a4b3-f85d89cabc58 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:48.028890 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 965/1962] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:47 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:48.925547 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2796e1bb-31ed-41e9-9a43-562e9144a9f7 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:48.926080 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2796e1bb-31ed-41e9-9a43-562e9144a9f7 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:48.926080 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2796e1bb-31ed-41e9-9a43-562e9144a9f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:48.955784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-2796e1bb-31ed-41e9-9a43-562e9144a9f7 None None] MFA Rules not processed for user `1e34111a3b2f4c1fb75252c4806d3da4`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:48.957068 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-2796e1bb-31ed-41e9-9a43-562e9144a9f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:49.027464 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 998/1963] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:48 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:49.038545 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fe57cf-3fbe-46a5-a4b3-f85d89cabc58 service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:49.040888 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-87fe57cf-3fbe-46a5-a4b3-f85d89cabc58 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:49.111166 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fe57cf-3fbe-46a5-a4b3-f85d89cabc58 service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:49.115377 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c955b8-e28a-4490-b8e6-010d0ad4b4aa service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:49.116019 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c955b8-e28a-4490-b8e6-010d0ad4b4aa service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:49.116306 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c955b8-e28a-4490-b8e6-010d0ad4b4aa service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:49.116585 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c955b8-e28a-4490-b8e6-010d0ad4b4aa service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:49.126646 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c955b8-e28a-4490-b8e6-010d0ad4b4aa service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:49.128143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c955b8-e28a-4490-b8e6-010d0ad4b4aa service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:49.138721 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 966/1964] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:49 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:50.569354 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9267dc26-1e50-45f4-bd07-170d8a123eef None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:50.569896 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9267dc26-1e50-45f4-bd07-170d8a123eef None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:50.569896 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9267dc26-1e50-45f4-bd07-170d8a123eef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:50.589691 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-9267dc26-1e50-45f4-bd07-170d8a123eef None None] MFA Rules not processed for user `1e34111a3b2f4c1fb75252c4806d3da4`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:50.590271 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-9267dc26-1e50-45f4-bd07-170d8a123eef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:50.603331 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 999/1965] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:50 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:51.702842 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c955b8-e28a-4490-b8e6-010d0ad4b4aa service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:51.704347 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-f3c955b8-e28a-4490-b8e6-010d0ad4b4aa service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:51.734572 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c955b8-e28a-4490-b8e6-010d0ad4b4aa service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:51.743910 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af9839f-b6ea-4a98-83d1-1f0e0fcfd47f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:51.745090 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af9839f-b6ea-4a98-83d1-1f0e0fcfd47f admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:51.745695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af9839f-b6ea-4a98-83d1-1f0e0fcfd47f admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:51.746223 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af9839f-b6ea-4a98-83d1-1f0e0fcfd47f admin admin] PATH_INFO: `/v3/users/eb357d729fa2427fab76907920e8fd2e` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:51.750187 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af9839f-b6ea-4a98-83d1-1f0e0fcfd47f admin admin] RBAC: Authorizing `identity:delete_user(user_id=eb357d729fa2427fab76907920e8fd2e)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:51.752812 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af9839f-b6ea-4a98-83d1-1f0e0fcfd47f admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:51.846508 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-1af9839f-b6ea-4a98-83d1-1f0e0fcfd47f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eb357d729fa2427fab76907920e8fd2e'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:51.847258 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-1af9839f-b6ea-4a98-83d1-1f0e0fcfd47f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eb357d729fa2427fab76907920e8fd2e'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:51.848650 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:51.859854 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-1af9839f-b6ea-4a98-83d1-1f0e0fcfd47f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eb357d729fa2427fab76907920e8fd2e'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:51.872693 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-1af9839f-b6ea-4a98-83d1-1f0e0fcfd47f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eb357d729fa2427fab76907920e8fd2e'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:51.873474 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-1af9839f-b6ea-4a98-83d1-1f0e0fcfd47f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eb357d729fa2427fab76907920e8fd2e'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:51.889079 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 967/1966] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:47:51 2022] DELETE /identity/v3/users/eb357d729fa2427fab76907920e8fd2e => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:47:51.893645 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8b26a4-4422-4bdf-90de-2af2dc07d712 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:51.894112 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8b26a4-4422-4bdf-90de-2af2dc07d712 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:51.894185 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8b26a4-4422-4bdf-90de-2af2dc07d712 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:51.915025 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-eb8b26a4-4422-4bdf-90de-2af2dc07d712 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:51.917852 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-eb8b26a4-4422-4bdf-90de-2af2dc07d712 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:51.951567 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:51.981123 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1000/1967] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:51 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:51.990893 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af9839f-b6ea-4a98-83d1-1f0e0fcfd47f admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:51.992384 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-1af9839f-b6ea-4a98-83d1-1f0e0fcfd47f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:52.051059 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af9839f-b6ea-4a98-83d1-1f0e0fcfd47f admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:52.055183 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8025ae-e8cc-4145-a95c-db389532a044 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:52.055832 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8025ae-e8cc-4145-a95c-db389532a044 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:52.056115 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8025ae-e8cc-4145-a95c-db389532a044 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:52.056389 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8025ae-e8cc-4145-a95c-db389532a044 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:52.065328 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c8025ae-e8cc-4145-a95c-db389532a044 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:52.066752 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c8025ae-e8cc-4145-a95c-db389532a044 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:52.077381 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 968/1968] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:51 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:47:52.349549 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8b26a4-4422-4bdf-90de-2af2dc07d712 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:52.351128 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-eb8b26a4-4422-4bdf-90de-2af2dc07d712 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:52.364759 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8b26a4-4422-4bdf-90de-2af2dc07d712 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:52.370133 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552b21dd-2ceb-47cb-8d59-56f40ea5639a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:52.370787 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552b21dd-2ceb-47cb-8d59-56f40ea5639a admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:52.371144 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552b21dd-2ceb-47cb-8d59-56f40ea5639a admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:52.371354 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552b21dd-2ceb-47cb-8d59-56f40ea5639a admin admin] PATH_INFO: `/v3/projects/4153c9ca4b2d4c6caf990f13aa45352a` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:52.372576 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552b21dd-2ceb-47cb-8d59-56f40ea5639a admin admin] RBAC: Authorizing `identity:delete_project(project_id=4153c9ca4b2d4c6caf990f13aa45352a)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:52.375290 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552b21dd-2ceb-47cb-8d59-56f40ea5639a admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:52.424528 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-552b21dd-2ceb-47cb-8d59-56f40ea5639a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4153c9ca4b2d4c6caf990f13aa45352a', 'request_id': 'req-552b21dd-2ceb-47cb-8d59-56f40ea5639a'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:52.431183 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-552b21dd-2ceb-47cb-8d59-56f40ea5639a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4153c9ca4b2d4c6caf990f13aa45352a', 'request_id': 'req-552b21dd-2ceb-47cb-8d59-56f40ea5639a'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:52.436738 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-552b21dd-2ceb-47cb-8d59-56f40ea5639a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4153c9ca4b2d4c6caf990f13aa45352a', 'request_id': 'req-552b21dd-2ceb-47cb-8d59-56f40ea5639a'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:52.450513 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-552b21dd-2ceb-47cb-8d59-56f40ea5639a admin admin] The token cache is being invalidate because project 4153c9ca4b2d4c6caf990f13aa45352a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:47:52.450807 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-552b21dd-2ceb-47cb-8d59-56f40ea5639a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:52.451446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-552b21dd-2ceb-47cb-8d59-56f40ea5639a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:47:52.453170 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1001/1969] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:47:52 2022] DELETE /identity/v3/projects/4153c9ca4b2d4c6caf990f13aa45352a => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 15:47:52.457933 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130a898f-6df3-449a-b758-8069b9952c33 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:52.458220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130a898f-6df3-449a-b758-8069b9952c33 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:52.458497 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130a898f-6df3-449a-b758-8069b9952c33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:52.478834 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-130a898f-6df3-449a-b758-8069b9952c33 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:52.480392 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-130a898f-6df3-449a-b758-8069b9952c33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:52.545367 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 969/1970] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:52 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:52.552913 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9091650f-e74e-4910-a149-a92937e503df None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:52.553233 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9091650f-e74e-4910-a149-a92937e503df None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:52.553511 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9091650f-e74e-4910-a149-a92937e503df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:52.575422 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-9091650f-e74e-4910-a149-a92937e503df None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:52.576809 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-9091650f-e74e-4910-a149-a92937e503df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:52.585346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1002/1971] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:52 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:47:52.591049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130a898f-6df3-449a-b758-8069b9952c33 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:52.601239 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130a898f-6df3-449a-b758-8069b9952c33 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:52.606909 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b384958-f1da-4e54-befb-fe989b385abb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:52.607693 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b384958-f1da-4e54-befb-fe989b385abb admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:52.607928 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b384958-f1da-4e54-befb-fe989b385abb admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:52.608195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b384958-f1da-4e54-befb-fe989b385abb admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:52.608842 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b384958-f1da-4e54-befb-fe989b385abb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:47:52.609144 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b384958-f1da-4e54-befb-fe989b385abb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:52.610879 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b384958-f1da-4e54-befb-fe989b385abb admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:52.626688 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 970/1972] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:47:52 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:47:52.627883 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9091650f-e74e-4910-a149-a92937e503df None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:52.638025 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9091650f-e74e-4910-a149-a92937e503df None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:52.642803 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf2b86f-4b2e-4b03-b990-d7a14ddba1f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:52.643404 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf2b86f-4b2e-4b03-b990-d7a14ddba1f9 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:52.643719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf2b86f-4b2e-4b03-b990-d7a14ddba1f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:52.643994 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf2b86f-4b2e-4b03-b990-d7a14ddba1f9 admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:52.644637 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf2b86f-4b2e-4b03-b990-d7a14ddba1f9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:52.646651 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf2b86f-4b2e-4b03-b990-d7a14ddba1f9 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:52.676696 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1003/1973] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:47:52 2022] POST /identity/v3/projects => generated 366 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:52.681226 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b384958-f1da-4e54-befb-fe989b385abb admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:52.691270 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b384958-f1da-4e54-befb-fe989b385abb admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:52.743106 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=96,keystone:DELETE=3,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:52.750527 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515e79c9-fc82-4e1a-b13d-386ee69cfc05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:52.751130 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515e79c9-fc82-4e1a-b13d-386ee69cfc05 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:52.751414 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515e79c9-fc82-4e1a-b13d-386ee69cfc05 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:52.751805 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515e79c9-fc82-4e1a-b13d-386ee69cfc05 admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:52.752354 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515e79c9-fc82-4e1a-b13d-386ee69cfc05 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:52.754940 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515e79c9-fc82-4e1a-b13d-386ee69cfc05 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:52.780457 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 971/1974] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:47:52 2022] POST /identity/v3/users => generated 284 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:47:52.781615 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf2b86f-4b2e-4b03-b990-d7a14ddba1f9 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:52.792558 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf2b86f-4b2e-4b03-b990-d7a14ddba1f9 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:52.797447 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19fad8f9-9125-409e-910d-69438cec9fa7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:52.798049 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fad8f9-9125-409e-910d-69438cec9fa7 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:52.798330 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fad8f9-9125-409e-910d-69438cec9fa7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:52.798604 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fad8f9-9125-409e-910d-69438cec9fa7 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:52.799293 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19fad8f9-9125-409e-910d-69438cec9fa7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:52.800899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19fad8f9-9125-409e-910d-69438cec9fa7 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:52.812061 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1004/1975] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:52 2022] GET /identity/v3/roles => generated 1427 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:52.815908 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515e79c9-fc82-4e1a-b13d-386ee69cfc05 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:52.827114 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515e79c9-fc82-4e1a-b13d-386ee69cfc05 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:52.835695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90eec4a-9b35-453d-85e7-c98ca832463b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:52.836834 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90eec4a-9b35-453d-85e7-c98ca832463b admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:52.837108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90eec4a-9b35-453d-85e7-c98ca832463b admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:52.837374 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90eec4a-9b35-453d-85e7-c98ca832463b admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:52.838375 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a90eec4a-9b35-453d-85e7-c98ca832463b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:52.840012 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a90eec4a-9b35-453d-85e7-c98ca832463b admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:52.852291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 972/1976] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:47:52 2022] GET /identity/v3/roles => generated 1427 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:47:52.856504 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19fad8f9-9125-409e-910d-69438cec9fa7 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:52.868740 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19fad8f9-9125-409e-910d-69438cec9fa7 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:52.873961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba858f40-b5ac-4909-9045-523fe0c872e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:52.874707 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba858f40-b5ac-4909-9045-523fe0c872e6 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:52.875022 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba858f40-b5ac-4909-9045-523fe0c872e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:52.875302 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba858f40-b5ac-4909-9045-523fe0c872e6 admin admin] PATH_INFO: `/v3/projects/234a63a9132d4e908be0a928e25acd40/users/0b0053d3167e435db8b9a9be0beabfce/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:52.891659 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba858f40-b5ac-4909-9045-523fe0c872e6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=234a63a9132d4e908be0a928e25acd40, user_id=0b0053d3167e435db8b9a9be0beabfce, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:52.894367 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba858f40-b5ac-4909-9045-523fe0c872e6 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:52.909654 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1005/1977] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:47:52 2022] PUT /identity/v3/projects/234a63a9132d4e908be0a928e25acd40/users/0b0053d3167e435db8b9a9be0beabfce/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 15:47:52.921255 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2319d0-fcb0-42fb-8ce5-59a7f36312fe None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:52.921579 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2319d0-fcb0-42fb-8ce5-59a7f36312fe None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:52.921821 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2319d0-fcb0-42fb-8ce5-59a7f36312fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:52.952584 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-fc2319d0-fcb0-42fb-8ce5-59a7f36312fe None None] MFA Rules not processed for user `0b0053d3167e435db8b9a9be0beabfce`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:47:52.953859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-fc2319d0-fcb0-42fb-8ce5-59a7f36312fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:53.026390 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 973/1978] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:47:52 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:47:53.037734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba858f40-b5ac-4909-9045-523fe0c872e6 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:47:53.040060 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-ba858f40-b5ac-4909-9045-523fe0c872e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:47:53.108343 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba858f40-b5ac-4909-9045-523fe0c872e6 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:47:53.112076 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309e98fd-1abd-4661-a5a8-36a6bb661911 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:47:53.112689 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309e98fd-1abd-4661-a5a8-36a6bb661911 service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:47:53.112974 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309e98fd-1abd-4661-a5a8-36a6bb661911 service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:47:53.113252 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309e98fd-1abd-4661-a5a8-36a6bb661911 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:47:53.123143 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-309e98fd-1abd-4661-a5a8-36a6bb661911 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:47:53.124737 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-309e98fd-1abd-4661-a5a8-36a6bb661911 service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:47:53.129594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:47:53.135967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1006/1979] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:47:53 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:48:03.022693 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=41,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:03.513433 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2319d0-fcb0-42fb-8ce5-59a7f36312fe None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:03.516211 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-fc2319d0-fcb0-42fb-8ce5-59a7f36312fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:03.622696 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2319d0-fcb0-42fb-8ce5-59a7f36312fe None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:03.627515 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26429444-7994-46ce-9a62-1b2759f6e8a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:03.628289 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26429444-7994-46ce-9a62-1b2759f6e8a2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:03.628646 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26429444-7994-46ce-9a62-1b2759f6e8a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:03.629533 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26429444-7994-46ce-9a62-1b2759f6e8a2 admin admin] PATH_INFO: `/v3/users/28da55f3d8da4bc486c210bb198249f4` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:03.630764 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26429444-7994-46ce-9a62-1b2759f6e8a2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=28da55f3d8da4bc486c210bb198249f4)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:03.633393 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26429444-7994-46ce-9a62-1b2759f6e8a2 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:03.701345 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-26429444-7994-46ce-9a62-1b2759f6e8a2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '28da55f3d8da4bc486c210bb198249f4'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:03.702366 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-26429444-7994-46ce-9a62-1b2759f6e8a2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '28da55f3d8da4bc486c210bb198249f4'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:03.715029 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-26429444-7994-46ce-9a62-1b2759f6e8a2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '28da55f3d8da4bc486c210bb198249f4'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:03.730143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-26429444-7994-46ce-9a62-1b2759f6e8a2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '28da55f3d8da4bc486c210bb198249f4'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:03.730740 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-26429444-7994-46ce-9a62-1b2759f6e8a2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '28da55f3d8da4bc486c210bb198249f4'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:03.749928 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 974/1980] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:48:03 2022] DELETE /identity/v3/users/28da55f3d8da4bc486c210bb198249f4 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:03.754597 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d026ea5-25c3-424c-b3aa-813434d05faa None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:03.755141 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d026ea5-25c3-424c-b3aa-813434d05faa None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:03.755141 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d026ea5-25c3-424c-b3aa-813434d05faa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:03.776163 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-5d026ea5-25c3-424c-b3aa-813434d05faa None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:03.777720 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-5d026ea5-25c3-424c-b3aa-813434d05faa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:03.858328 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1007/1981] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:03 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:03.892488 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26429444-7994-46ce-9a62-1b2759f6e8a2 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:03.894987 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-26429444-7994-46ce-9a62-1b2759f6e8a2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:03.988588 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26429444-7994-46ce-9a62-1b2759f6e8a2 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:03.994291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32ebb7d-b53e-4583-86c0-94d75a68d363 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:03.994918 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32ebb7d-b53e-4583-86c0-94d75a68d363 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:03.995199 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32ebb7d-b53e-4583-86c0-94d75a68d363 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:03.995474 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32ebb7d-b53e-4583-86c0-94d75a68d363 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:04.009363 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32ebb7d-b53e-4583-86c0-94d75a68d363 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:04.010958 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32ebb7d-b53e-4583-86c0-94d75a68d363 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:04.026895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 975/1982] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:48:03 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:48:04.985410 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d026ea5-25c3-424c-b3aa-813434d05faa None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:04.987480 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-5d026ea5-25c3-424c-b3aa-813434d05faa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:05.008354 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d026ea5-25c3-424c-b3aa-813434d05faa None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:05.013920 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91afd60-c856-43a5-ace3-b8dde5e74952 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:05.014901 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91afd60-c856-43a5-ace3-b8dde5e74952 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:05.015397 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91afd60-c856-43a5-ace3-b8dde5e74952 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:05.015977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91afd60-c856-43a5-ace3-b8dde5e74952 admin admin] PATH_INFO: `/v3/users/4881ba284a8e481bab0452ab8527c0aa` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:05.017640 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91afd60-c856-43a5-ace3-b8dde5e74952 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4881ba284a8e481bab0452ab8527c0aa)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:05.020871 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91afd60-c856-43a5-ace3-b8dde5e74952 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:05.122837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32ebb7d-b53e-4583-86c0-94d75a68d363 service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:05.123720 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-c91afd60-c856-43a5-ace3-b8dde5e74952 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4881ba284a8e481bab0452ab8527c0aa'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:05.124785 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-e32ebb7d-b53e-4583-86c0-94d75a68d363 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:05.125477 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-c91afd60-c856-43a5-ace3-b8dde5e74952 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4881ba284a8e481bab0452ab8527c0aa'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:05.126278 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-c91afd60-c856-43a5-ace3-b8dde5e74952 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4881ba284a8e481bab0452ab8527c0aa'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:05.136745 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-c91afd60-c856-43a5-ace3-b8dde5e74952 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4881ba284a8e481bab0452ab8527c0aa'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:05.152884 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32ebb7d-b53e-4583-86c0-94d75a68d363 service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:05.164875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccaa27c5-40f4-4fed-8741-029c2180427f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:05.165550 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccaa27c5-40f4-4fed-8741-029c2180427f admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:05.165951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccaa27c5-40f4-4fed-8741-029c2180427f admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:05.166325 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccaa27c5-40f4-4fed-8741-029c2180427f admin admin] PATH_INFO: `/v3/projects/9df6fd89c8854f55b0c274b7563b664b` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:05.168383 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccaa27c5-40f4-4fed-8741-029c2180427f admin admin] RBAC: Authorizing `identity:delete_project(project_id=9df6fd89c8854f55b0c274b7563b664b)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:05.171935 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccaa27c5-40f4-4fed-8741-029c2180427f admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:05.176297 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-c91afd60-c856-43a5-ace3-b8dde5e74952 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4881ba284a8e481bab0452ab8527c0aa'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:05.201502 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1008/1983] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:48:04 2022] DELETE /identity/v3/users/4881ba284a8e481bab0452ab8527c0aa => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:05.208079 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682b9b17-f07d-43c1-86ec-537cc07e4d8e None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:05.208979 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682b9b17-f07d-43c1-86ec-537cc07e4d8e None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:05.209566 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682b9b17-f07d-43c1-86ec-537cc07e4d8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:05.224942 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:05.264418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-682b9b17-f07d-43c1-86ec-537cc07e4d8e None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:05.266830 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-682b9b17-f07d-43c1-86ec-537cc07e4d8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:05.293217 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-ccaa27c5-40f4-4fed-8741-029c2180427f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9df6fd89c8854f55b0c274b7563b664b', 'request_id': 'req-ccaa27c5-40f4-4fed-8741-029c2180427f'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:05.304245 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-ccaa27c5-40f4-4fed-8741-029c2180427f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9df6fd89c8854f55b0c274b7563b664b', 'request_id': 'req-ccaa27c5-40f4-4fed-8741-029c2180427f'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:05.322007 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-ccaa27c5-40f4-4fed-8741-029c2180427f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9df6fd89c8854f55b0c274b7563b664b', 'request_id': 'req-ccaa27c5-40f4-4fed-8741-029c2180427f'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:05.341969 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-ccaa27c5-40f4-4fed-8741-029c2180427f admin admin] The token cache is being invalidate because project 9df6fd89c8854f55b0c274b7563b664b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:48:05.342310 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-ccaa27c5-40f4-4fed-8741-029c2180427f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:05.345490 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-ccaa27c5-40f4-4fed-8741-029c2180427f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:05.347738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 976/1984] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:48:05 2022] DELETE /identity/v3/projects/9df6fd89c8854f55b0c274b7563b664b => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:05.352672 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccaa27c5-40f4-4fed-8741-029c2180427f admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:05.355260 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-ccaa27c5-40f4-4fed-8741-029c2180427f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:05.432369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1009/1985] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:05 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:05.436916 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682b9b17-f07d-43c1-86ec-537cc07e4d8e None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:05.439420 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-682b9b17-f07d-43c1-86ec-537cc07e4d8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:05.459086 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccaa27c5-40f4-4fed-8741-029c2180427f admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:05.465507 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f8a312-1ecf-44d3-a328-7118b14c4612 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:05.466287 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:05.466439 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f8a312-1ecf-44d3-a328-7118b14c4612 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:05.467491 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f8a312-1ecf-44d3-a328-7118b14c4612 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:05.469607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f8a312-1ecf-44d3-a328-7118b14c4612 admin admin] PATH_INFO: `/v3/users/ddde15b3f78242a689a9a05b82fbc9d0` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:05.470446 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f8a312-1ecf-44d3-a328-7118b14c4612 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ddde15b3f78242a689a9a05b82fbc9d0)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:05.472618 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f8a312-1ecf-44d3-a328-7118b14c4612 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:05.560808 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682b9b17-f07d-43c1-86ec-537cc07e4d8e None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:05.565124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c58b8c-edb5-4826-9882-8c25f7b5fea3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:05.566085 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c58b8c-edb5-4826-9882-8c25f7b5fea3 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:05.566539 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c58b8c-edb5-4826-9882-8c25f7b5fea3 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:05.567131 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c58b8c-edb5-4826-9882-8c25f7b5fea3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:05.572059 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-24f8a312-1ecf-44d3-a328-7118b14c4612 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ddde15b3f78242a689a9a05b82fbc9d0'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:05.572923 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-24f8a312-1ecf-44d3-a328-7118b14c4612 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ddde15b3f78242a689a9a05b82fbc9d0'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:05.583821 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c58b8c-edb5-4826-9882-8c25f7b5fea3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:05.584441 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-24f8a312-1ecf-44d3-a328-7118b14c4612 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ddde15b3f78242a689a9a05b82fbc9d0'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:05.587969 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c58b8c-edb5-4826-9882-8c25f7b5fea3 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:05.600168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1010/1986] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:48:05 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 15:48:05.604125 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-24f8a312-1ecf-44d3-a328-7118b14c4612 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ddde15b3f78242a689a9a05b82fbc9d0'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:05.605092 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-24f8a312-1ecf-44d3-a328-7118b14c4612 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ddde15b3f78242a689a9a05b82fbc9d0'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:05.623420 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 977/1987] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:48:05 2022] DELETE /identity/v3/users/ddde15b3f78242a689a9a05b82fbc9d0 => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 15:48:06.118827 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c58b8c-edb5-4826-9882-8c25f7b5fea3 service neutron] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:06.120532 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-91c58b8c-edb5-4826-9882-8c25f7b5fea3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:06.136455 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c58b8c-edb5-4826-9882-8c25f7b5fea3 service neutron] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:06.144274 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f8a312-1ecf-44d3-a328-7118b14c4612 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:06.146458 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-24f8a312-1ecf-44d3-a328-7118b14c4612 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:06.189093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:06.206718 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd34c50-33df-420f-8a2e-6464b570ebd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:06.207367 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd34c50-33df-420f-8a2e-6464b570ebd6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:06.207782 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd34c50-33df-420f-8a2e-6464b570ebd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:06.208476 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd34c50-33df-420f-8a2e-6464b570ebd6 admin admin] PATH_INFO: `/v3/projects/6d781073258242179398fe7e35565d37` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:06.209798 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd34c50-33df-420f-8a2e-6464b570ebd6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6d781073258242179398fe7e35565d37)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:06.213200 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd34c50-33df-420f-8a2e-6464b570ebd6 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:06.235754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f8a312-1ecf-44d3-a328-7118b14c4612 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:06.240497 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65dddfc0-04a9-45f7-bd8a-90b79be5404b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:06.241162 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dddfc0-04a9-45f7-bd8a-90b79be5404b admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:06.241509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dddfc0-04a9-45f7-bd8a-90b79be5404b admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:06.242354 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dddfc0-04a9-45f7-bd8a-90b79be5404b admin admin] PATH_INFO: `/v3/projects/39d3d22eb8a945abb61dd8a082a3b57f` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:06.243903 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65dddfc0-04a9-45f7-bd8a-90b79be5404b admin admin] RBAC: Authorizing `identity:delete_project(project_id=39d3d22eb8a945abb61dd8a082a3b57f)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:06.246168 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65dddfc0-04a9-45f7-bd8a-90b79be5404b admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:06.264181 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-6dd34c50-33df-420f-8a2e-6464b570ebd6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6d781073258242179398fe7e35565d37', 'request_id': 'req-6dd34c50-33df-420f-8a2e-6464b570ebd6'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:06.270783 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-6dd34c50-33df-420f-8a2e-6464b570ebd6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6d781073258242179398fe7e35565d37', 'request_id': 'req-6dd34c50-33df-420f-8a2e-6464b570ebd6'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:06.277251 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-6dd34c50-33df-420f-8a2e-6464b570ebd6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6d781073258242179398fe7e35565d37', 'request_id': 'req-6dd34c50-33df-420f-8a2e-6464b570ebd6'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:06.290661 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-6dd34c50-33df-420f-8a2e-6464b570ebd6 admin admin] The token cache is being invalidate because project 6d781073258242179398fe7e35565d37 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:48:06.291045 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-6dd34c50-33df-420f-8a2e-6464b570ebd6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:06.291747 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-6dd34c50-33df-420f-8a2e-6464b570ebd6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:06.293418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1011/1988] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:48:06 2022] DELETE /identity/v3/projects/6d781073258242179398fe7e35565d37 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 15:48:06.297236 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd34c50-33df-420f-8a2e-6464b570ebd6 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:06.297885 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-65dddfc0-04a9-45f7-bd8a-90b79be5404b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '39d3d22eb8a945abb61dd8a082a3b57f', 'request_id': 'req-65dddfc0-04a9-45f7-bd8a-90b79be5404b'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:06.298785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-6dd34c50-33df-420f-8a2e-6464b570ebd6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:06.305074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-65dddfc0-04a9-45f7-bd8a-90b79be5404b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '39d3d22eb8a945abb61dd8a082a3b57f', 'request_id': 'req-65dddfc0-04a9-45f7-bd8a-90b79be5404b'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:06.311768 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-65dddfc0-04a9-45f7-bd8a-90b79be5404b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '39d3d22eb8a945abb61dd8a082a3b57f', 'request_id': 'req-65dddfc0-04a9-45f7-bd8a-90b79be5404b'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:06.325191 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-65dddfc0-04a9-45f7-bd8a-90b79be5404b admin admin] The token cache is being invalidate because project 39d3d22eb8a945abb61dd8a082a3b57f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:48:06.325479 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-65dddfc0-04a9-45f7-bd8a-90b79be5404b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:06.326135 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-65dddfc0-04a9-45f7-bd8a-90b79be5404b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:06.327695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 978/1989] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:48:06 2022] DELETE /identity/v3/projects/39d3d22eb8a945abb61dd8a082a3b57f => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 15:48:06.332293 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb3edf7-6cce-46cc-858e-80a86970c96b None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:06.332582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb3edf7-6cce-46cc-858e-80a86970c96b None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:06.332858 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb3edf7-6cce-46cc-858e-80a86970c96b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:06.352895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-2cb3edf7-6cce-46cc-858e-80a86970c96b None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:06.353927 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:06.354247 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-2cb3edf7-6cce-46cc-858e-80a86970c96b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:06.371694 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd34c50-33df-420f-8a2e-6464b570ebd6 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:06.417332 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 979/1990] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:06 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:48:06.425883 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5caffc-eb4f-4e41-aeeb-5ba26d20a60c None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:06.426170 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5caffc-eb4f-4e41-aeeb-5ba26d20a60c None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:06.426441 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5caffc-eb4f-4e41-aeeb-5ba26d20a60c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:06.434160 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec33c96f-db4d-4d3f-b9c3-a07f3e4f0bdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:06.434806 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec33c96f-db4d-4d3f-b9c3-a07f3e4f0bdc admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:06.435090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec33c96f-db4d-4d3f-b9c3-a07f3e4f0bdc admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:06.435458 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec33c96f-db4d-4d3f-b9c3-a07f3e4f0bdc admin admin] PATH_INFO: `/v3/users/079bbec14aaf48678e62d829e31ffa67` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:06.436655 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec33c96f-db4d-4d3f-b9c3-a07f3e4f0bdc admin admin] RBAC: Authorizing `identity:delete_user(user_id=079bbec14aaf48678e62d829e31ffa67)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:06.439435 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec33c96f-db4d-4d3f-b9c3-a07f3e4f0bdc admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:06.446727 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-5d5caffc-eb4f-4e41-aeeb-5ba26d20a60c None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:06.447946 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-5d5caffc-eb4f-4e41-aeeb-5ba26d20a60c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:06.456334 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 980/1991] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:06 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:48:06.460345 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5caffc-eb4f-4e41-aeeb-5ba26d20a60c None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:06.470365 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5caffc-eb4f-4e41-aeeb-5ba26d20a60c None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:06.476681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7affeaa1-bc34-4536-828d-a55c7d46e7c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:06.477520 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7affeaa1-bc34-4536-828d-a55c7d46e7c0 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:06.477742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7affeaa1-bc34-4536-828d-a55c7d46e7c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:06.478071 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7affeaa1-bc34-4536-828d-a55c7d46e7c0 admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:06.478783 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7affeaa1-bc34-4536-828d-a55c7d46e7c0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:48:06.479089 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7affeaa1-bc34-4536-828d-a55c7d46e7c0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:06.480724 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7affeaa1-bc34-4536-828d-a55c7d46e7c0 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:06.489437 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-ec33c96f-db4d-4d3f-b9c3-a07f3e4f0bdc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '079bbec14aaf48678e62d829e31ffa67'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:06.490187 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-ec33c96f-db4d-4d3f-b9c3-a07f3e4f0bdc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '079bbec14aaf48678e62d829e31ffa67'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:06.490806 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-ec33c96f-db4d-4d3f-b9c3-a07f3e4f0bdc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '079bbec14aaf48678e62d829e31ffa67'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:06.493545 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 981/1992] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:48:06 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:48:06.498122 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7affeaa1-bc34-4536-828d-a55c7d46e7c0 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:06.499367 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-ec33c96f-db4d-4d3f-b9c3-a07f3e4f0bdc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '079bbec14aaf48678e62d829e31ffa67'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:06.499968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-7affeaa1-bc34-4536-828d-a55c7d46e7c0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:06.512296 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-ec33c96f-db4d-4d3f-b9c3-a07f3e4f0bdc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '079bbec14aaf48678e62d829e31ffa67'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:06.513427 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7affeaa1-bc34-4536-828d-a55c7d46e7c0 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:06.518626 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8483a253-74d0-4b15-b87b-e9b5020d68b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:06.519280 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8483a253-74d0-4b15-b87b-e9b5020d68b8 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:06.519560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8483a253-74d0-4b15-b87b-e9b5020d68b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:06.519860 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8483a253-74d0-4b15-b87b-e9b5020d68b8 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:06.520538 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8483a253-74d0-4b15-b87b-e9b5020d68b8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:06.522466 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8483a253-74d0-4b15-b87b-e9b5020d68b8 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:06.525862 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1012/1993] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:48:06 2022] DELETE /identity/v3/users/079bbec14aaf48678e62d829e31ffa67 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 15:48:06.561051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 982/1994] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:48:06 2022] POST /identity/v3/projects => generated 384 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 15:48:06.562475 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec33c96f-db4d-4d3f-b9c3-a07f3e4f0bdc admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:06.631599 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec33c96f-db4d-4d3f-b9c3-a07f3e4f0bdc admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:06.649510 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:06.700573 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5128779-957d-46e7-b074-9ac1aeceb314 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:06.701194 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5128779-957d-46e7-b074-9ac1aeceb314 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:06.701473 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5128779-957d-46e7-b074-9ac1aeceb314 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:06.701746 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5128779-957d-46e7-b074-9ac1aeceb314 admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:06.702423 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5128779-957d-46e7-b074-9ac1aeceb314 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:06.704594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5128779-957d-46e7-b074-9ac1aeceb314 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:06.727005 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1013/1995] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:48:06 2022] POST /identity/v3/users => generated 293 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:48:06.731291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8483a253-74d0-4b15-b87b-e9b5020d68b8 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:06.742699 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8483a253-74d0-4b15-b87b-e9b5020d68b8 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:06.747536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ceeb74a-6688-46e2-9298-b588e45c016e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:06.748239 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ceeb74a-6688-46e2-9298-b588e45c016e admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:06.748615 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ceeb74a-6688-46e2-9298-b588e45c016e admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:06.748899 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ceeb74a-6688-46e2-9298-b588e45c016e admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:06.749553 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ceeb74a-6688-46e2-9298-b588e45c016e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:06.751157 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ceeb74a-6688-46e2-9298-b588e45c016e admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:06.763864 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 983/1996] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:48:06 2022] GET /identity/v3/roles => generated 1427 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 15:48:06.768320 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5128779-957d-46e7-b074-9ac1aeceb314 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:06.809254 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5128779-957d-46e7-b074-9ac1aeceb314 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:06.814371 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb73ac87-01df-4019-82b8-eac7ba8d0829 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:06.816387 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb73ac87-01df-4019-82b8-eac7ba8d0829 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:06.816677 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb73ac87-01df-4019-82b8-eac7ba8d0829 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:06.818626 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb73ac87-01df-4019-82b8-eac7ba8d0829 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:06.819490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb73ac87-01df-4019-82b8-eac7ba8d0829 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:06.825512 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb73ac87-01df-4019-82b8-eac7ba8d0829 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:06.864795 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ceeb74a-6688-46e2-9298-b588e45c016e admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:06.866759 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-6ceeb74a-6688-46e2-9298-b588e45c016e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:06.882266 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:06.893876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1014/1997] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:48:06 2022] GET /identity/v3/roles => generated 1427 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:48:06.894829 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb73ac87-01df-4019-82b8-eac7ba8d0829 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:06.907231 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ceeb74a-6688-46e2-9298-b588e45c016e admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:06.915535 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65aac788-ff0e-44c1-9c3d-fd0983ff26fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:06.917659 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65aac788-ff0e-44c1-9c3d-fd0983ff26fb admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:06.918316 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65aac788-ff0e-44c1-9c3d-fd0983ff26fb admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:06.918926 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65aac788-ff0e-44c1-9c3d-fd0983ff26fb admin admin] PATH_INFO: `/v3/projects/275cbfa2c18248f0b6d2289bd0906465` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:06.921191 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65aac788-ff0e-44c1-9c3d-fd0983ff26fb admin admin] RBAC: Authorizing `identity:delete_project(project_id=275cbfa2c18248f0b6d2289bd0906465)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:06.925118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65aac788-ff0e-44c1-9c3d-fd0983ff26fb admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:06.954422 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb73ac87-01df-4019-82b8-eac7ba8d0829 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:06.962062 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e242f4ac-2eff-465b-9789-927de65b578e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:06.962360 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e242f4ac-2eff-465b-9789-927de65b578e admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:06.962731 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e242f4ac-2eff-465b-9789-927de65b578e admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:06.963375 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e242f4ac-2eff-465b-9789-927de65b578e admin admin] PATH_INFO: `/v3/projects/f53c407c80ac4aafa3f2054926f0f162/users/8b6576bd776340bf80354b80e81686e1/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:06.994824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e242f4ac-2eff-465b-9789-927de65b578e admin admin] RBAC: Authorizing `identity:create_grant(project_id=f53c407c80ac4aafa3f2054926f0f162, user_id=8b6576bd776340bf80354b80e81686e1, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:06.997762 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e242f4ac-2eff-465b-9789-927de65b578e admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:07.017016 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-65aac788-ff0e-44c1-9c3d-fd0983ff26fb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '275cbfa2c18248f0b6d2289bd0906465', 'request_id': 'req-65aac788-ff0e-44c1-9c3d-fd0983ff26fb'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:07.021153 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1015/1998] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:48:06 2022] PUT /identity/v3/projects/f53c407c80ac4aafa3f2054926f0f162/users/8b6576bd776340bf80354b80e81686e1/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:07.031870 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e86fa2-2edb-488e-9fe6-0477b02de1f5 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:07.032147 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e86fa2-2edb-488e-9fe6-0477b02de1f5 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:07.032426 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e86fa2-2edb-488e-9fe6-0477b02de1f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:07.038427 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-65aac788-ff0e-44c1-9c3d-fd0983ff26fb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '275cbfa2c18248f0b6d2289bd0906465', 'request_id': 'req-65aac788-ff0e-44c1-9c3d-fd0983ff26fb'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:07.049058 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-65aac788-ff0e-44c1-9c3d-fd0983ff26fb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '275cbfa2c18248f0b6d2289bd0906465', 'request_id': 'req-65aac788-ff0e-44c1-9c3d-fd0983ff26fb'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:07.064099 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-23e86fa2-2edb-488e-9fe6-0477b02de1f5 None None] MFA Rules not processed for user `8b6576bd776340bf80354b80e81686e1`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:07.065384 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-23e86fa2-2edb-488e-9fe6-0477b02de1f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:07.068864 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-65aac788-ff0e-44c1-9c3d-fd0983ff26fb admin admin] The token cache is being invalidate because project 275cbfa2c18248f0b6d2289bd0906465 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:48:07.069105 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-65aac788-ff0e-44c1-9c3d-fd0983ff26fb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:07.069753 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-65aac788-ff0e-44c1-9c3d-fd0983ff26fb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:07.072556 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 984/1999] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:48:06 2022] DELETE /identity/v3/projects/275cbfa2c18248f0b6d2289bd0906465 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 15:48:07.077255 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594e85b3-6d84-4475-b665-6a63abfa1345 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:07.077547 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594e85b3-6d84-4475-b665-6a63abfa1345 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:07.077865 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594e85b3-6d84-4475-b665-6a63abfa1345 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:07.101841 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-594e85b3-6d84-4475-b665-6a63abfa1345 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:07.103707 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-594e85b3-6d84-4475-b665-6a63abfa1345 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:07.155821 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1016/2000] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:07 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:48:07.160149 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e86fa2-2edb-488e-9fe6-0477b02de1f5 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:07.161534 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-23e86fa2-2edb-488e-9fe6-0477b02de1f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:07.168476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 985/2001] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:07 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:07.176344 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400bcea6-dc76-4e7b-bce8-07a71afde91f None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:07.176525 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e86fa2-2edb-488e-9fe6-0477b02de1f5 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:07.176759 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400bcea6-dc76-4e7b-bce8-07a71afde91f None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:07.177046 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400bcea6-dc76-4e7b-bce8-07a71afde91f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:07.182451 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853e008a-66f4-4b1b-afab-c49040c9bf98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:07.183064 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853e008a-66f4-4b1b-afab-c49040c9bf98 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:07.183347 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853e008a-66f4-4b1b-afab-c49040c9bf98 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:07.183617 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853e008a-66f4-4b1b-afab-c49040c9bf98 admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:07.184467 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-853e008a-66f4-4b1b-afab-c49040c9bf98 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:07.186375 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-853e008a-66f4-4b1b-afab-c49040c9bf98 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:07.200054 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-400bcea6-dc76-4e7b-bce8-07a71afde91f None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:07.201266 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-400bcea6-dc76-4e7b-bce8-07a71afde91f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:07.209777 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 986/2002] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:07 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:48:07.214202 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400bcea6-dc76-4e7b-bce8-07a71afde91f None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:07.219057 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1017/2003] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:48:07 2022] POST /identity/v3/projects => generated 382 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 15:48:07.222917 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853e008a-66f4-4b1b-afab-c49040c9bf98 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:07.223877 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400bcea6-dc76-4e7b-bce8-07a71afde91f None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:07.271266 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:07.292872 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853e008a-66f4-4b1b-afab-c49040c9bf98 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:07.293283 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e11125-04d2-4ce0-97c9-701d57f8d6c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:07.294156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e11125-04d2-4ce0-97c9-701d57f8d6c9 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:07.294507 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e11125-04d2-4ce0-97c9-701d57f8d6c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:07.294695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e11125-04d2-4ce0-97c9-701d57f8d6c9 admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:07.295347 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e11125-04d2-4ce0-97c9-701d57f8d6c9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:48:07.295679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e11125-04d2-4ce0-97c9-701d57f8d6c9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:07.297282 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e11125-04d2-4ce0-97c9-701d57f8d6c9 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:07.298247 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e691d075-8df6-42fe-a693-a517e1cc2fd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:07.299040 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e691d075-8df6-42fe-a693-a517e1cc2fd4 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:07.299534 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e691d075-8df6-42fe-a693-a517e1cc2fd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:07.300040 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e691d075-8df6-42fe-a693-a517e1cc2fd4 admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:07.300712 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e691d075-8df6-42fe-a693-a517e1cc2fd4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:07.302701 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e691d075-8df6-42fe-a693-a517e1cc2fd4 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:07.309772 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 987/2004] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:48:07 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:48:07.314139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e11125-04d2-4ce0-97c9-701d57f8d6c9 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:07.326030 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e11125-04d2-4ce0-97c9-701d57f8d6c9 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:07.327130 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1018/2005] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:48:07 2022] POST /identity/v3/users => generated 298 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:48:07.330779 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7079f39d-3389-4783-9b0a-46ada01fd2c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:07.330910 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e691d075-8df6-42fe-a693-a517e1cc2fd4 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:07.331560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7079f39d-3389-4783-9b0a-46ada01fd2c5 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:07.331878 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7079f39d-3389-4783-9b0a-46ada01fd2c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:07.332151 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7079f39d-3389-4783-9b0a-46ada01fd2c5 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:07.332785 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7079f39d-3389-4783-9b0a-46ada01fd2c5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:07.334680 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7079f39d-3389-4783-9b0a-46ada01fd2c5 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:07.341766 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e691d075-8df6-42fe-a693-a517e1cc2fd4 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:07.346448 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a2f219-af77-4ae0-893d-653badeb3cba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:07.346626 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:07.347049 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a2f219-af77-4ae0-893d-653badeb3cba admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:07.347330 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a2f219-af77-4ae0-893d-653badeb3cba admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:07.347674 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a2f219-af77-4ae0-893d-653badeb3cba admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:07.348625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a2f219-af77-4ae0-893d-653badeb3cba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:07.350219 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a2f219-af77-4ae0-893d-653badeb3cba admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:07.362077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1019/2006] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:48:07 2022] GET /identity/v3/roles => generated 1427 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 15:48:07.369579 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 988/2007] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:48:07 2022] POST /identity/v3/projects => generated 368 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 15:48:07.369757 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a2f219-af77-4ae0-893d-653badeb3cba admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:07.370502 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7079f39d-3389-4783-9b0a-46ada01fd2c5 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:07.377219 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a2f219-af77-4ae0-893d-653badeb3cba admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:07.431520 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7079f39d-3389-4783-9b0a-46ada01fd2c5 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:07.435697 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c334664b-ffb3-4a2b-b756-cf2413910e23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:07.436186 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94773cd5-3e35-444c-8c32-8700a714f64b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:07.436361 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c334664b-ffb3-4a2b-b756-cf2413910e23 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:07.436642 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c334664b-ffb3-4a2b-b756-cf2413910e23 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:07.436782 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94773cd5-3e35-444c-8c32-8700a714f64b admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:07.436925 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c334664b-ffb3-4a2b-b756-cf2413910e23 admin admin] PATH_INFO: `/v3/projects/38344d5fc1e04b0da08643a182adc503/users/4ce6999c66934c548e9735e6913a1b40/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:07.437070 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94773cd5-3e35-444c-8c32-8700a714f64b admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:07.437320 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94773cd5-3e35-444c-8c32-8700a714f64b admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:07.437938 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94773cd5-3e35-444c-8c32-8700a714f64b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:07.439876 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94773cd5-3e35-444c-8c32-8700a714f64b admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:07.452437 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c334664b-ffb3-4a2b-b756-cf2413910e23 admin admin] RBAC: Authorizing `identity:create_grant(project_id=38344d5fc1e04b0da08643a182adc503, user_id=4ce6999c66934c548e9735e6913a1b40, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:07.455870 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c334664b-ffb3-4a2b-b756-cf2413910e23 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:07.463128 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 989/2008] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:48:07 2022] POST /identity/v3/users => generated 285 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 15:48:07.467406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94773cd5-3e35-444c-8c32-8700a714f64b admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:07.470016 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1020/2009] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:48:07 2022] PUT /identity/v3/projects/38344d5fc1e04b0da08643a182adc503/users/4ce6999c66934c548e9735e6913a1b40/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 15:48:07.474039 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c334664b-ffb3-4a2b-b756-cf2413910e23 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:07.479305 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94773cd5-3e35-444c-8c32-8700a714f64b admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:07.541093 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c334664b-ffb3-4a2b-b756-cf2413910e23 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:07.542723 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19425741-b884-4d20-b113-1fb7b2daa45c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:07.543338 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19425741-b884-4d20-b113-1fb7b2daa45c admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:07.543612 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19425741-b884-4d20-b113-1fb7b2daa45c admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:07.543973 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19425741-b884-4d20-b113-1fb7b2daa45c admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:07.544580 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19425741-b884-4d20-b113-1fb7b2daa45c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:07.546073 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57be4cac-2534-4bef-8dc8-a56c7f1b739c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:07.546198 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19425741-b884-4d20-b113-1fb7b2daa45c admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:07.546709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57be4cac-2534-4bef-8dc8-a56c7f1b739c admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:07.546995 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57be4cac-2534-4bef-8dc8-a56c7f1b739c admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:07.547277 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57be4cac-2534-4bef-8dc8-a56c7f1b739c admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:07.547960 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57be4cac-2534-4bef-8dc8-a56c7f1b739c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:07.549617 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57be4cac-2534-4bef-8dc8-a56c7f1b739c admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:07.557816 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 990/2010] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:48:07 2022] GET /identity/v3/roles => generated 1427 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 15:48:07.561530 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1021/2011] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:48:07 2022] GET /identity/v3/roles => generated 1427 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 15:48:07.561811 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57be4cac-2534-4bef-8dc8-a56c7f1b739c admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:07.564734 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19425741-b884-4d20-b113-1fb7b2daa45c admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:07.572814 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57be4cac-2534-4bef-8dc8-a56c7f1b739c admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:07.576261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19425741-b884-4d20-b113-1fb7b2daa45c admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:07.577535 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e38cc8-c820-4a6a-8fcd-d3de12cf0b14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:07.578134 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e38cc8-c820-4a6a-8fcd-d3de12cf0b14 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:07.578420 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e38cc8-c820-4a6a-8fcd-d3de12cf0b14 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:07.578701 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e38cc8-c820-4a6a-8fcd-d3de12cf0b14 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:07.579453 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e38cc8-c820-4a6a-8fcd-d3de12cf0b14 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:07.580990 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c40c4f-de06-46cc-9261-7462969c79d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:07.581134 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e38cc8-c820-4a6a-8fcd-d3de12cf0b14 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:07.581633 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c40c4f-de06-46cc-9261-7462969c79d5 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:07.581915 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c40c4f-de06-46cc-9261-7462969c79d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:07.582295 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c40c4f-de06-46cc-9261-7462969c79d5 admin admin] PATH_INFO: `/v3/projects/38344d5fc1e04b0da08643a182adc503/users/4ce6999c66934c548e9735e6913a1b40/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:07.584098 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c40c4f-de06-46cc-9261-7462969c79d5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=38344d5fc1e04b0da08643a182adc503, user_id=4ce6999c66934c548e9735e6913a1b40, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:07.586784 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c40c4f-de06-46cc-9261-7462969c79d5 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:07.591403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1022/2012] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:48:07 2022] GET /identity/v3/roles => generated 1427 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 15:48:07.595275 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e38cc8-c820-4a6a-8fcd-d3de12cf0b14 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:07.597390 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 991/2013] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:48:07 2022] PUT /identity/v3/projects/38344d5fc1e04b0da08643a182adc503/users/4ce6999c66934c548e9735e6913a1b40/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 15:48:07.605277 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e38cc8-c820-4a6a-8fcd-d3de12cf0b14 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:07.607532 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891ccc11-a4fc-4a8f-8b35-8bfee4581cd8 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:07.607860 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891ccc11-a4fc-4a8f-8b35-8bfee4581cd8 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:07.608136 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891ccc11-a4fc-4a8f-8b35-8bfee4581cd8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:07.634340 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:07.638319 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-891ccc11-a4fc-4a8f-8b35-8bfee4581cd8 None None] MFA Rules not processed for user `4ce6999c66934c548e9735e6913a1b40`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:07.639540 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-891ccc11-a4fc-4a8f-8b35-8bfee4581cd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:07.667829 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3c6f45-85eb-45a2-be8d-3e98a20aaffa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:07.668506 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3c6f45-85eb-45a2-be8d-3e98a20aaffa admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:07.668794 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3c6f45-85eb-45a2-be8d-3e98a20aaffa admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:07.669075 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3c6f45-85eb-45a2-be8d-3e98a20aaffa admin admin] PATH_INFO: `/v3/projects/42c23107db424deca6201f783b8ed8c9/users/a885042923d24cd29166cbace4c08097/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:07.685973 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c3c6f45-85eb-45a2-be8d-3e98a20aaffa admin admin] RBAC: Authorizing `identity:create_grant(project_id=42c23107db424deca6201f783b8ed8c9, user_id=a885042923d24cd29166cbace4c08097, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:07.688677 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c3c6f45-85eb-45a2-be8d-3e98a20aaffa admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:07.691843 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:07.702414 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1023/2014] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:48:07 2022] PUT /identity/v3/projects/42c23107db424deca6201f783b8ed8c9/users/a885042923d24cd29166cbace4c08097/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 15:48:07.713901 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 992/2015] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:07 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:48:07.714423 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7ed4a1-074b-40b1-a434-7a17171d94e9 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:07.714774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7ed4a1-074b-40b1-a434-7a17171d94e9 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:07.715055 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7ed4a1-074b-40b1-a434-7a17171d94e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:07.724507 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891ccc11-a4fc-4a8f-8b35-8bfee4581cd8 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:07.725881 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-891ccc11-a4fc-4a8f-8b35-8bfee4581cd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:07.744943 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-6b7ed4a1-074b-40b1-a434-7a17171d94e9 None None] MFA Rules not processed for user `a885042923d24cd29166cbace4c08097`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:07.746165 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-6b7ed4a1-074b-40b1-a434-7a17171d94e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:07.787367 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891ccc11-a4fc-4a8f-8b35-8bfee4581cd8 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:07.813683 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1024/2016] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:07 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:07.818108 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7ed4a1-074b-40b1-a434-7a17171d94e9 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:07.840732 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c40d10-848f-4472-ab8f-90a8c25b3218 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:07.841637 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c40d10-848f-4472-ab8f-90a8c25b3218 service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:07.841637 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c40d10-848f-4472-ab8f-90a8c25b3218 service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:07.841902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c40d10-848f-4472-ab8f-90a8c25b3218 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:07.892236 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7ed4a1-074b-40b1-a434-7a17171d94e9 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:07.896963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7555628-37a7-4529-b6ad-f7edd9d5d432 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:07.897571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7555628-37a7-4529-b6ad-f7edd9d5d432 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:07.897848 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7555628-37a7-4529-b6ad-f7edd9d5d432 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:07.898143 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7555628-37a7-4529-b6ad-f7edd9d5d432 admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:07.898891 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7555628-37a7-4529-b6ad-f7edd9d5d432 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:07.900951 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7555628-37a7-4529-b6ad-f7edd9d5d432 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:07.917301 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c40d10-848f-4472-ab8f-90a8c25b3218 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:07.918774 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c40d10-848f-4472-ab8f-90a8c25b3218 service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:07.931523 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 993/2017] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:48:07 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 207 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 15:48:07.937111 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1025/2018] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:48:07 2022] POST /identity/v3/projects => generated 370 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:48:07.940600 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c40d10-848f-4472-ab8f-90a8c25b3218 service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:07.958632 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c40d10-848f-4472-ab8f-90a8c25b3218 service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:08.024705 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:08.034979 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb2247e-36a8-40a8-bf7d-9c41308bd347 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:08.035604 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb2247e-36a8-40a8-bf7d-9c41308bd347 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:08.035912 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb2247e-36a8-40a8-bf7d-9c41308bd347 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:08.036335 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb2247e-36a8-40a8-bf7d-9c41308bd347 admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:08.037117 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb2247e-36a8-40a8-bf7d-9c41308bd347 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:08.039596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb2247e-36a8-40a8-bf7d-9c41308bd347 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:08.043727 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7555628-37a7-4529-b6ad-f7edd9d5d432 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:08.078856 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:08.080340 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 994/2019] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:48:07 2022] POST /identity/v3/users => generated 292 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:48:08.084299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb2247e-36a8-40a8-bf7d-9c41308bd347 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:08.095746 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb2247e-36a8-40a8-bf7d-9c41308bd347 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:08.100926 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a8a2a2-d7c9-46eb-9f31-2afae6917dbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:08.102059 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a8a2a2-d7c9-46eb-9f31-2afae6917dbd admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:08.102337 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a8a2a2-d7c9-46eb-9f31-2afae6917dbd admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:08.102606 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a8a2a2-d7c9-46eb-9f31-2afae6917dbd admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:08.103254 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a8a2a2-d7c9-46eb-9f31-2afae6917dbd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:08.105040 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a8a2a2-d7c9-46eb-9f31-2afae6917dbd admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:08.119427 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7555628-37a7-4529-b6ad-f7edd9d5d432 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:08.121682 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 995/2020] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:48:08 2022] GET /identity/v3/roles => generated 1427 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 15:48:08.122657 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a8a2a2-d7c9-46eb-9f31-2afae6917dbd admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:08.125503 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f06044-34a5-4f52-a45b-0381f236eeff service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:08.126296 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f06044-34a5-4f52-a45b-0381f236eeff service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:08.126582 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f06044-34a5-4f52-a45b-0381f236eeff service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:08.126867 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f06044-34a5-4f52-a45b-0381f236eeff service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:08.136098 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a8a2a2-d7c9-46eb-9f31-2afae6917dbd admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:08.141202 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e231f644-79e1-4217-abca-7489bcff5554 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:08.142061 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e231f644-79e1-4217-abca-7489bcff5554 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:08.142339 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e231f644-79e1-4217-abca-7489bcff5554 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:08.142634 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e231f644-79e1-4217-abca-7489bcff5554 admin admin] PATH_INFO: `/v3/projects/6558ebaf165c41bca42094829ed0c4ec/users/cc27ecf3c87d4c4895506d3753702dfd/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:08.161886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e231f644-79e1-4217-abca-7489bcff5554 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6558ebaf165c41bca42094829ed0c4ec, user_id=cc27ecf3c87d4c4895506d3753702dfd, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:08.165491 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e231f644-79e1-4217-abca-7489bcff5554 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:08.185220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 996/2021] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:48:08 2022] PUT /identity/v3/projects/6558ebaf165c41bca42094829ed0c4ec/users/cc27ecf3c87d4c4895506d3753702dfd/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:08.190740 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e231f644-79e1-4217-abca-7489bcff5554 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:08.247464 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f06044-34a5-4f52-a45b-0381f236eeff service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:08.249017 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f06044-34a5-4f52-a45b-0381f236eeff service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:08.261685 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1026/2022] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:48:08 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 218 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 15:48:08.297976 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e231f644-79e1-4217-abca-7489bcff5554 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:08.304073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51787d31-fe0e-4c25-ad00-e34d4bd295da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:08.305355 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51787d31-fe0e-4c25-ad00-e34d4bd295da admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:08.305595 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51787d31-fe0e-4c25-ad00-e34d4bd295da admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:08.305885 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51787d31-fe0e-4c25-ad00-e34d4bd295da admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:08.306612 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51787d31-fe0e-4c25-ad00-e34d4bd295da admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:08.308314 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51787d31-fe0e-4c25-ad00-e34d4bd295da admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:08.319813 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 997/2023] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:48:08 2022] GET /identity/v3/roles => generated 1427 bytes in 131 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 15:48:08.324329 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f06044-34a5-4f52-a45b-0381f236eeff service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:08.338069 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f06044-34a5-4f52-a45b-0381f236eeff service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:08.354214 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07318f1-df24-489d-b080-cc6954c30b14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:08.355331 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07318f1-df24-489d-b080-cc6954c30b14 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:08.355758 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07318f1-df24-489d-b080-cc6954c30b14 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:08.356299 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07318f1-df24-489d-b080-cc6954c30b14 admin admin] PATH_INFO: `/v3/projects/6558ebaf165c41bca42094829ed0c4ec/users/cc27ecf3c87d4c4895506d3753702dfd/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:08.366636 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07318f1-df24-489d-b080-cc6954c30b14 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6558ebaf165c41bca42094829ed0c4ec, user_id=cc27ecf3c87d4c4895506d3753702dfd, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:08.372023 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07318f1-df24-489d-b080-cc6954c30b14 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:08.388871 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1027/2024] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:48:08 2022] PUT /identity/v3/projects/6558ebaf165c41bca42094829ed0c4ec/users/cc27ecf3c87d4c4895506d3753702dfd/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:08.399501 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f396cc4-f53c-4479-b97e-f62ef284be49 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:08.399904 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f396cc4-f53c-4479-b97e-f62ef284be49 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:08.400187 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f396cc4-f53c-4479-b97e-f62ef284be49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:08.438048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-3f396cc4-f53c-4479-b97e-f62ef284be49 None None] MFA Rules not processed for user `cc27ecf3c87d4c4895506d3753702dfd`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:08.439390 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-3f396cc4-f53c-4479-b97e-f62ef284be49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:08.543977 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 998/2025] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:08 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:08.557007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07318f1-df24-489d-b080-cc6954c30b14 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:08.559073 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-a07318f1-df24-489d-b080-cc6954c30b14 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:08.631804 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07318f1-df24-489d-b080-cc6954c30b14 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:08.636898 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e93bcd-dc3d-46c7-a70a-7b83ccdd8dff service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:08.637666 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e93bcd-dc3d-46c7-a70a-7b83ccdd8dff service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:08.638054 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e93bcd-dc3d-46c7-a70a-7b83ccdd8dff service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:08.638429 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e93bcd-dc3d-46c7-a70a-7b83ccdd8dff service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:08.651708 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e93bcd-dc3d-46c7-a70a-7b83ccdd8dff service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:08.653388 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e93bcd-dc3d-46c7-a70a-7b83ccdd8dff service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:08.666197 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1028/2026] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:48:08 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:48:08.720981 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f396cc4-f53c-4479-b97e-f62ef284be49 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:08.732157 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f396cc4-f53c-4479-b97e-f62ef284be49 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:08.735847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121b38c9-8960-4e77-a298-891d3024ac6d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:08.736487 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121b38c9-8960-4e77-a298-891d3024ac6d service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:08.736775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121b38c9-8960-4e77-a298-891d3024ac6d service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:08.737054 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121b38c9-8960-4e77-a298-891d3024ac6d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:08.769810 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:08.805484 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121b38c9-8960-4e77-a298-891d3024ac6d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:08.806399 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121b38c9-8960-4e77-a298-891d3024ac6d service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:08.815582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 999/2027] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:48:08 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:48:10.978181 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e93bcd-dc3d-46c7-a70a-7b83ccdd8dff service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:11.056537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e93bcd-dc3d-46c7-a70a-7b83ccdd8dff service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:11.061418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10cb6f45-cb19-4ebb-9b59-0338ac913339 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:11.062080 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10cb6f45-cb19-4ebb-9b59-0338ac913339 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:11.062362 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10cb6f45-cb19-4ebb-9b59-0338ac913339 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:11.062642 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10cb6f45-cb19-4ebb-9b59-0338ac913339 admin admin] PATH_INFO: `/v3/users/8b6576bd776340bf80354b80e81686e1` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:11.063823 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10cb6f45-cb19-4ebb-9b59-0338ac913339 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8b6576bd776340bf80354b80e81686e1)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:11.065772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10cb6f45-cb19-4ebb-9b59-0338ac913339 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:11.082739 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=98,keystone:INSERT=1,keystone:DELETE=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:11.128470 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-10cb6f45-cb19-4ebb-9b59-0338ac913339 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8b6576bd776340bf80354b80e81686e1'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:11.129132 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-10cb6f45-cb19-4ebb-9b59-0338ac913339 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8b6576bd776340bf80354b80e81686e1'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:11.129742 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-10cb6f45-cb19-4ebb-9b59-0338ac913339 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8b6576bd776340bf80354b80e81686e1'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:11.139234 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-10cb6f45-cb19-4ebb-9b59-0338ac913339 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8b6576bd776340bf80354b80e81686e1'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:11.156239 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-10cb6f45-cb19-4ebb-9b59-0338ac913339 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8b6576bd776340bf80354b80e81686e1'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:11.170470 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1029/2028] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:48:10 2022] DELETE /identity/v3/users/8b6576bd776340bf80354b80e81686e1 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:11.176103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef064d4-64c0-45c9-8c6b-f6dd5df50427 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:11.176633 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef064d4-64c0-45c9-8c6b-f6dd5df50427 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:11.176900 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef064d4-64c0-45c9-8c6b-f6dd5df50427 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:11.199401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-9ef064d4-64c0-45c9-8c6b-f6dd5df50427 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:11.200746 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-9ef064d4-64c0-45c9-8c6b-f6dd5df50427 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:11.264302 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1000/2029] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:11 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:11.274354 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10cb6f45-cb19-4ebb-9b59-0338ac913339 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:11.279016 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-10cb6f45-cb19-4ebb-9b59-0338ac913339 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:11.344484 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10cb6f45-cb19-4ebb-9b59-0338ac913339 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:11.348456 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35bb1404-1694-45d8-b7b6-4b4ccd275e02 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:11.349070 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35bb1404-1694-45d8-b7b6-4b4ccd275e02 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:11.349357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35bb1404-1694-45d8-b7b6-4b4ccd275e02 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:11.350162 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35bb1404-1694-45d8-b7b6-4b4ccd275e02 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:11.362424 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35bb1404-1694-45d8-b7b6-4b4ccd275e02 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:11.363983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35bb1404-1694-45d8-b7b6-4b4ccd275e02 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:11.377517 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1030/2030] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:48:11 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:48:11.687797 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef064d4-64c0-45c9-8c6b-f6dd5df50427 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:11.689463 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-9ef064d4-64c0-45c9-8c6b-f6dd5df50427 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:11.714607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef064d4-64c0-45c9-8c6b-f6dd5df50427 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:11.721213 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4cbe6b-f199-453c-9d83-477e6101403c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:11.722100 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4cbe6b-f199-453c-9d83-477e6101403c admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:11.722995 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4cbe6b-f199-453c-9d83-477e6101403c admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:11.722995 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4cbe6b-f199-453c-9d83-477e6101403c admin admin] PATH_INFO: `/v3/projects/f53c407c80ac4aafa3f2054926f0f162` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:11.724097 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a4cbe6b-f199-453c-9d83-477e6101403c admin admin] RBAC: Authorizing `identity:delete_project(project_id=f53c407c80ac4aafa3f2054926f0f162)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:11.726452 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a4cbe6b-f199-453c-9d83-477e6101403c admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:11.792869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-1a4cbe6b-f199-453c-9d83-477e6101403c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f53c407c80ac4aafa3f2054926f0f162', 'request_id': 'req-1a4cbe6b-f199-453c-9d83-477e6101403c'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:11.799618 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-1a4cbe6b-f199-453c-9d83-477e6101403c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f53c407c80ac4aafa3f2054926f0f162', 'request_id': 'req-1a4cbe6b-f199-453c-9d83-477e6101403c'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:11.813465 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-1a4cbe6b-f199-453c-9d83-477e6101403c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f53c407c80ac4aafa3f2054926f0f162', 'request_id': 'req-1a4cbe6b-f199-453c-9d83-477e6101403c'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:11.840807 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-1a4cbe6b-f199-453c-9d83-477e6101403c admin admin] The token cache is being invalidate because project f53c407c80ac4aafa3f2054926f0f162 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:48:11.841095 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-1a4cbe6b-f199-453c-9d83-477e6101403c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:11.841896 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-1a4cbe6b-f199-453c-9d83-477e6101403c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:11.846659 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1001/2031] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:48:11 2022] DELETE /identity/v3/projects/f53c407c80ac4aafa3f2054926f0f162 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 15:48:11.847669 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35bb1404-1694-45d8-b7b6-4b4ccd275e02 service neutron] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:11.857793 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35bb1404-1694-45d8-b7b6-4b4ccd275e02 service neutron] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:11.926069 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f7009c-5a2b-49d5-aa7c-a38db34df82f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:11.926779 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f7009c-5a2b-49d5-aa7c-a38db34df82f admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:11.927036 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f7009c-5a2b-49d5-aa7c-a38db34df82f admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:11.927248 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f7009c-5a2b-49d5-aa7c-a38db34df82f admin admin] PATH_INFO: `/v3/users/4ce6999c66934c548e9735e6913a1b40` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:11.928624 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f7009c-5a2b-49d5-aa7c-a38db34df82f admin admin] RBAC: Authorizing `identity:delete_user(user_id=4ce6999c66934c548e9735e6913a1b40)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:11.932069 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f7009c-5a2b-49d5-aa7c-a38db34df82f admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:12.008702 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:12.032668 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-18f7009c-5a2b-49d5-aa7c-a38db34df82f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4ce6999c66934c548e9735e6913a1b40'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:12.033564 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-18f7009c-5a2b-49d5-aa7c-a38db34df82f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4ce6999c66934c548e9735e6913a1b40'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:12.034495 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-18f7009c-5a2b-49d5-aa7c-a38db34df82f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4ce6999c66934c548e9735e6913a1b40'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:12.046396 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-18f7009c-5a2b-49d5-aa7c-a38db34df82f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4ce6999c66934c548e9735e6913a1b40'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:12.070938 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-18f7009c-5a2b-49d5-aa7c-a38db34df82f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4ce6999c66934c548e9735e6913a1b40'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:12.087514 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1031/2032] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:48:11 2022] DELETE /identity/v3/users/4ce6999c66934c548e9735e6913a1b40 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:12.341072 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4cbe6b-f199-453c-9d83-477e6101403c admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:12.342964 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-1a4cbe6b-f199-453c-9d83-477e6101403c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:12.356532 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4cbe6b-f199-453c-9d83-477e6101403c admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:12.414756 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f7009c-5a2b-49d5-aa7c-a38db34df82f admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:12.416657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-18f7009c-5a2b-49d5-aa7c-a38db34df82f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:12.418181 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:12.436063 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e99ae8-fc95-4891-b398-985049e90fc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:12.437039 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e99ae8-fc95-4891-b398-985049e90fc5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:12.437514 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e99ae8-fc95-4891-b398-985049e90fc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:12.437983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e99ae8-fc95-4891-b398-985049e90fc5 admin admin] PATH_INFO: `/v3/users/a885042923d24cd29166cbace4c08097` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:12.440258 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e99ae8-fc95-4891-b398-985049e90fc5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a885042923d24cd29166cbace4c08097)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:12.442549 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e99ae8-fc95-4891-b398-985049e90fc5 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:12.496006 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-86e99ae8-fc95-4891-b398-985049e90fc5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a885042923d24cd29166cbace4c08097'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:12.496673 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-86e99ae8-fc95-4891-b398-985049e90fc5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a885042923d24cd29166cbace4c08097'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:12.499591 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f7009c-5a2b-49d5-aa7c-a38db34df82f admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:12.503464 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-86e99ae8-fc95-4891-b398-985049e90fc5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a885042923d24cd29166cbace4c08097'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:12.504262 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831020b4-29b3-4d20-92a8-a0de46b92852 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:12.504895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831020b4-29b3-4d20-92a8-a0de46b92852 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:12.505270 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831020b4-29b3-4d20-92a8-a0de46b92852 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:12.505555 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831020b4-29b3-4d20-92a8-a0de46b92852 admin admin] PATH_INFO: `/v3/projects/38344d5fc1e04b0da08643a182adc503` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:12.506700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831020b4-29b3-4d20-92a8-a0de46b92852 admin admin] RBAC: Authorizing `identity:delete_project(project_id=38344d5fc1e04b0da08643a182adc503)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:12.509929 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831020b4-29b3-4d20-92a8-a0de46b92852 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:12.517801 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-86e99ae8-fc95-4891-b398-985049e90fc5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a885042923d24cd29166cbace4c08097'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:12.518653 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-86e99ae8-fc95-4891-b398-985049e90fc5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a885042923d24cd29166cbace4c08097'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:12.533180 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1002/2033] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:48:12 2022] DELETE /identity/v3/users/a885042923d24cd29166cbace4c08097 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:12.537450 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf6884b-b572-4091-849c-7f5a65addcc9 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:12.537734 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf6884b-b572-4091-849c-7f5a65addcc9 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:12.538006 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf6884b-b572-4091-849c-7f5a65addcc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:12.558225 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-acf6884b-b572-4091-849c-7f5a65addcc9 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:12.559446 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-acf6884b-b572-4091-849c-7f5a65addcc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:12.560983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-831020b4-29b3-4d20-92a8-a0de46b92852 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '38344d5fc1e04b0da08643a182adc503', 'request_id': 'req-831020b4-29b3-4d20-92a8-a0de46b92852'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:12.567364 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-831020b4-29b3-4d20-92a8-a0de46b92852 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '38344d5fc1e04b0da08643a182adc503', 'request_id': 'req-831020b4-29b3-4d20-92a8-a0de46b92852'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:12.574061 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-831020b4-29b3-4d20-92a8-a0de46b92852 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '38344d5fc1e04b0da08643a182adc503', 'request_id': 'req-831020b4-29b3-4d20-92a8-a0de46b92852'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:12.588279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-831020b4-29b3-4d20-92a8-a0de46b92852 admin admin] The token cache is being invalidate because project 38344d5fc1e04b0da08643a182adc503 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:48:12.588844 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-831020b4-29b3-4d20-92a8-a0de46b92852 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:12.589222 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-831020b4-29b3-4d20-92a8-a0de46b92852 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:12.590916 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1032/2034] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:48:12 2022] DELETE /identity/v3/projects/38344d5fc1e04b0da08643a182adc503 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 15:48:12.595426 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3f2956-b1f0-440b-ba23-cbf9727b0df9 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:12.595805 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3f2956-b1f0-440b-ba23-cbf9727b0df9 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:12.596081 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3f2956-b1f0-440b-ba23-cbf9727b0df9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:12.616510 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-dd3f2956-b1f0-440b-ba23-cbf9727b0df9 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:12.617756 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-dd3f2956-b1f0-440b-ba23-cbf9727b0df9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:12.622404 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1003/2035] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:12 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:48:12.625630 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1033/2036] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:12 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:48:12.632931 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf6884b-b572-4091-849c-7f5a65addcc9 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:12.634652 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-acf6884b-b572-4091-849c-7f5a65addcc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:12.635050 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea926918-83a1-4bc3-914e-6ad00c074996 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:12.635339 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea926918-83a1-4bc3-914e-6ad00c074996 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:12.635612 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea926918-83a1-4bc3-914e-6ad00c074996 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:12.655679 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-ea926918-83a1-4bc3-914e-6ad00c074996 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:12.656877 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-ea926918-83a1-4bc3-914e-6ad00c074996 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:12.665031 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1034/2037] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:12 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:12.669193 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea926918-83a1-4bc3-914e-6ad00c074996 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:12.678797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea926918-83a1-4bc3-914e-6ad00c074996 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:12.685145 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c43a9e-daf9-44f9-8784-b78a15250a05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:12.685769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c43a9e-daf9-44f9-8784-b78a15250a05 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:12.686161 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c43a9e-daf9-44f9-8784-b78a15250a05 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:12.686434 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c43a9e-daf9-44f9-8784-b78a15250a05 admin admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:12.687082 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c43a9e-daf9-44f9-8784-b78a15250a05 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:48:12.687384 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c43a9e-daf9-44f9-8784-b78a15250a05 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:12.689086 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c43a9e-daf9-44f9-8784-b78a15250a05 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:12.693483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf6884b-b572-4091-849c-7f5a65addcc9 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:12.698741 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306589b7-4ebf-4416-932c-85c3990d582e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:12.699350 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306589b7-4ebf-4416-932c-85c3990d582e service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:12.699650 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306589b7-4ebf-4416-932c-85c3990d582e service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:12.699939 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306589b7-4ebf-4416-932c-85c3990d582e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:12.702082 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1035/2038] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:48:12 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:48:12.706641 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c43a9e-daf9-44f9-8784-b78a15250a05 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:12.709994 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-306589b7-4ebf-4416-932c-85c3990d582e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:12.711322 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-306589b7-4ebf-4416-932c-85c3990d582e service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:12.717596 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c43a9e-daf9-44f9-8784-b78a15250a05 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:12.721187 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1004/2039] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:48:12 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:48:12.724864 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a71fa8-b2ce-47a8-a60b-b27bfb1b9dcf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:12.725545 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a71fa8-b2ce-47a8-a60b-b27bfb1b9dcf admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:12.725895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a71fa8-b2ce-47a8-a60b-b27bfb1b9dcf admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:12.726325 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a71fa8-b2ce-47a8-a60b-b27bfb1b9dcf admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:12.727216 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a71fa8-b2ce-47a8-a60b-b27bfb1b9dcf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:12.729133 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a71fa8-b2ce-47a8-a60b-b27bfb1b9dcf admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:12.746043 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:12.763423 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1036/2040] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:48:12 2022] POST /identity/v3/projects => generated 384 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 15:48:12.767353 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306589b7-4ebf-4416-932c-85c3990d582e service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:12.779394 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306589b7-4ebf-4416-932c-85c3990d582e service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:12.805839 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:12.870829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2385a25-b4e8-4f81-9627-5381d24d63fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:12.871669 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2385a25-b4e8-4f81-9627-5381d24d63fe admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:12.872087 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2385a25-b4e8-4f81-9627-5381d24d63fe admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:12.872381 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2385a25-b4e8-4f81-9627-5381d24d63fe admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:12.873098 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2385a25-b4e8-4f81-9627-5381d24d63fe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:12.875094 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2385a25-b4e8-4f81-9627-5381d24d63fe admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:12.903997 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1005/2041] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:48:12 2022] POST /identity/v3/users => generated 293 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:48:12.909350 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a71fa8-b2ce-47a8-a60b-b27bfb1b9dcf admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:12.923493 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a71fa8-b2ce-47a8-a60b-b27bfb1b9dcf admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:12.927509 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e238d604-7930-4446-9889-c83a665d5b85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:12.928308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e238d604-7930-4446-9889-c83a665d5b85 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:12.928652 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e238d604-7930-4446-9889-c83a665d5b85 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:12.929020 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e238d604-7930-4446-9889-c83a665d5b85 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:12.930950 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e238d604-7930-4446-9889-c83a665d5b85 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:12.932855 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e238d604-7930-4446-9889-c83a665d5b85 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:12.947871 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1037/2042] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:48:12 2022] GET /identity/v3/roles => generated 1427 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:48:12.949567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2385a25-b4e8-4f81-9627-5381d24d63fe admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:12.961466 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2385a25-b4e8-4f81-9627-5381d24d63fe admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:12.966977 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3456a27-889c-44b6-82ad-bcbc6cf87533 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:12.967746 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3456a27-889c-44b6-82ad-bcbc6cf87533 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:12.968103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3456a27-889c-44b6-82ad-bcbc6cf87533 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:12.968469 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3456a27-889c-44b6-82ad-bcbc6cf87533 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:12.969211 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3456a27-889c-44b6-82ad-bcbc6cf87533 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:12.973279 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3456a27-889c-44b6-82ad-bcbc6cf87533 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:12.994943 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1006/2043] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:48:12 2022] GET /identity/v3/roles => generated 1427 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 15:48:12.999366 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e238d604-7930-4446-9889-c83a665d5b85 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:13.013645 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e238d604-7930-4446-9889-c83a665d5b85 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:13.022509 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f23f467-d6dd-432f-9946-f2cb1d3e328f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:13.024098 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f23f467-d6dd-432f-9946-f2cb1d3e328f admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:13.024463 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f23f467-d6dd-432f-9946-f2cb1d3e328f admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:13.024837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f23f467-d6dd-432f-9946-f2cb1d3e328f admin admin] PATH_INFO: `/v3/projects/5b7bc49fffc74a2ba1c77dbc24710d5d/users/c18a557fc5484e85b6a3353ad944c133/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:13.044779 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f23f467-d6dd-432f-9946-f2cb1d3e328f admin admin] RBAC: Authorizing `identity:create_grant(project_id=5b7bc49fffc74a2ba1c77dbc24710d5d, user_id=c18a557fc5484e85b6a3353ad944c133, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:13.050017 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f23f467-d6dd-432f-9946-f2cb1d3e328f admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:13.081431 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1038/2044] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:48:12 2022] PUT /identity/v3/projects/5b7bc49fffc74a2ba1c77dbc24710d5d/users/c18a557fc5484e85b6a3353ad944c133/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:13.084323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfebefd-1273-4105-951d-3f624277e616 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:13.084617 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfebefd-1273-4105-951d-3f624277e616 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:13.084894 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfebefd-1273-4105-951d-3f624277e616 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:13.118915 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f23f467-d6dd-432f-9946-f2cb1d3e328f admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:13.120799 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-5f23f467-d6dd-432f-9946-f2cb1d3e328f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:13.146786 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-ddfebefd-1273-4105-951d-3f624277e616 None None] MFA Rules not processed for user `c18a557fc5484e85b6a3353ad944c133`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:13.148196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-ddfebefd-1273-4105-951d-3f624277e616 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:13.199936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f23f467-d6dd-432f-9946-f2cb1d3e328f admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:13.207318 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a1eead-f1d2-458d-8986-397fb3b95f85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:13.208209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a1eead-f1d2-458d-8986-397fb3b95f85 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:13.208669 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a1eead-f1d2-458d-8986-397fb3b95f85 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:13.209312 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a1eead-f1d2-458d-8986-397fb3b95f85 admin admin] PATH_INFO: `/v3/projects/42c23107db424deca6201f783b8ed8c9` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:13.211781 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42a1eead-f1d2-458d-8986-397fb3b95f85 admin admin] RBAC: Authorizing `identity:delete_project(project_id=42c23107db424deca6201f783b8ed8c9)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:13.216224 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42a1eead-f1d2-458d-8986-397fb3b95f85 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:13.265009 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1007/2045] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:13 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:13.269776 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfebefd-1273-4105-951d-3f624277e616 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:13.283588 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfebefd-1273-4105-951d-3f624277e616 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:13.291110 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69fd896-b10c-41e6-aa6f-aa9002d20a21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:13.291911 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69fd896-b10c-41e6-aa6f-aa9002d20a21 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:13.292379 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69fd896-b10c-41e6-aa6f-aa9002d20a21 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:13.292791 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69fd896-b10c-41e6-aa6f-aa9002d20a21 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:13.293548 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69fd896-b10c-41e6-aa6f-aa9002d20a21 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:13.296441 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69fd896-b10c-41e6-aa6f-aa9002d20a21 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:13.338931 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-42a1eead-f1d2-458d-8986-397fb3b95f85 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '42c23107db424deca6201f783b8ed8c9', 'request_id': 'req-42a1eead-f1d2-458d-8986-397fb3b95f85'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:13.345337 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1008/2046] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:48:13 2022] POST /identity/v3/projects => generated 384 bytes in 76 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:48:13.350490 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69fd896-b10c-41e6-aa6f-aa9002d20a21 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:13.358393 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-42a1eead-f1d2-458d-8986-397fb3b95f85 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '42c23107db424deca6201f783b8ed8c9', 'request_id': 'req-42a1eead-f1d2-458d-8986-397fb3b95f85'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:13.384552 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-42a1eead-f1d2-458d-8986-397fb3b95f85 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '42c23107db424deca6201f783b8ed8c9', 'request_id': 'req-42a1eead-f1d2-458d-8986-397fb3b95f85'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:13.415002 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-42a1eead-f1d2-458d-8986-397fb3b95f85 admin admin] The token cache is being invalidate because project 42c23107db424deca6201f783b8ed8c9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:48:13.415582 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-42a1eead-f1d2-458d-8986-397fb3b95f85 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:13.418192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-42a1eead-f1d2-458d-8986-397fb3b95f85 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:13.422309 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1039/2047] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:48:13 2022] DELETE /identity/v3/projects/42c23107db424deca6201f783b8ed8c9 => generated 0 bytes in 304 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:13.431128 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a1eead-f1d2-458d-8986-397fb3b95f85 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:13.432998 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-42a1eead-f1d2-458d-8986-397fb3b95f85 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:13.469931 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69fd896-b10c-41e6-aa6f-aa9002d20a21 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:13.476068 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c799a94-9ae1-4cc4-aef2-365eb67bc8c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:13.477163 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c799a94-9ae1-4cc4-aef2-365eb67bc8c4 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:13.477443 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c799a94-9ae1-4cc4-aef2-365eb67bc8c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:13.477729 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c799a94-9ae1-4cc4-aef2-365eb67bc8c4 admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:13.478356 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c799a94-9ae1-4cc4-aef2-365eb67bc8c4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:13.481348 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c799a94-9ae1-4cc4-aef2-365eb67bc8c4 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:13.498448 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:13.527006 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1009/2048] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:48:13 2022] POST /identity/v3/users => generated 299 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:48:13.533368 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c799a94-9ae1-4cc4-aef2-365eb67bc8c4 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:13.540735 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-0c799a94-9ae1-4cc4-aef2-365eb67bc8c4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:13.546315 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:INSERT=3,keystone:SELECT=90,keystone:DELETE=7 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:13.567550 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c799a94-9ae1-4cc4-aef2-365eb67bc8c4 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:13.574442 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618ff321-96c6-4591-83e1-02fc525bbfb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:13.575065 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618ff321-96c6-4591-83e1-02fc525bbfb3 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:13.575343 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618ff321-96c6-4591-83e1-02fc525bbfb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:13.575616 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618ff321-96c6-4591-83e1-02fc525bbfb3 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:13.576298 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-618ff321-96c6-4591-83e1-02fc525bbfb3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:13.579216 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-618ff321-96c6-4591-83e1-02fc525bbfb3 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:13.595324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1010/2049] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:48:13 2022] GET /identity/v3/roles => generated 1427 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:48:13.595519 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a1eead-f1d2-458d-8986-397fb3b95f85 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:13.605949 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618ff321-96c6-4591-83e1-02fc525bbfb3 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:13.615859 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff11522-cb5b-4af5-b7e3-97671f81c08a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:13.616656 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff11522-cb5b-4af5-b7e3-97671f81c08a admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:13.617755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff11522-cb5b-4af5-b7e3-97671f81c08a admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:13.618364 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff11522-cb5b-4af5-b7e3-97671f81c08a admin admin] PATH_INFO: `/v3/users/cc27ecf3c87d4c4895506d3753702dfd` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:13.619935 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff11522-cb5b-4af5-b7e3-97671f81c08a admin admin] RBAC: Authorizing `identity:delete_user(user_id=cc27ecf3c87d4c4895506d3753702dfd)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:13.622279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff11522-cb5b-4af5-b7e3-97671f81c08a admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:13.625205 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618ff321-96c6-4591-83e1-02fc525bbfb3 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:13.630986 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d010524-e4dc-4f74-85a4-9c9daff655ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:13.643382 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d010524-e4dc-4f74-85a4-9c9daff655ef admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:13.644135 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d010524-e4dc-4f74-85a4-9c9daff655ef admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:13.644296 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d010524-e4dc-4f74-85a4-9c9daff655ef admin admin] PATH_INFO: `/v3/projects/6c4299ab3e584f579073d3bc2047c764/users/049e22dc9adb4579b3f3fe8af7865ca4/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:13.666696 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d010524-e4dc-4f74-85a4-9c9daff655ef admin admin] RBAC: Authorizing `identity:create_grant(project_id=6c4299ab3e584f579073d3bc2047c764, user_id=049e22dc9adb4579b3f3fe8af7865ca4, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:13.669456 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d010524-e4dc-4f74-85a4-9c9daff655ef admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:13.695650 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1011/2050] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:48:13 2022] PUT /identity/v3/projects/6c4299ab3e584f579073d3bc2047c764/users/049e22dc9adb4579b3f3fe8af7865ca4/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:13.702446 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d010524-e4dc-4f74-85a4-9c9daff655ef admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:13.713198 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-3ff11522-cb5b-4af5-b7e3-97671f81c08a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cc27ecf3c87d4c4895506d3753702dfd'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:13.714933 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-3ff11522-cb5b-4af5-b7e3-97671f81c08a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cc27ecf3c87d4c4895506d3753702dfd'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:13.715517 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-3ff11522-cb5b-4af5-b7e3-97671f81c08a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cc27ecf3c87d4c4895506d3753702dfd'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:13.733743 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-3ff11522-cb5b-4af5-b7e3-97671f81c08a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cc27ecf3c87d4c4895506d3753702dfd'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:13.751591 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-3ff11522-cb5b-4af5-b7e3-97671f81c08a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cc27ecf3c87d4c4895506d3753702dfd'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:13.776157 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1040/2051] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:48:13 2022] DELETE /identity/v3/users/cc27ecf3c87d4c4895506d3753702dfd => generated 0 bytes in 344 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:13.821188 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d010524-e4dc-4f74-85a4-9c9daff655ef admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:13.910899 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2326b02e-f951-4439-bb66-4bdf95cb1e97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:13.919294 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2326b02e-f951-4439-bb66-4bdf95cb1e97 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:13.919294 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2326b02e-f951-4439-bb66-4bdf95cb1e97 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:13.919294 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2326b02e-f951-4439-bb66-4bdf95cb1e97 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:13.919294 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2326b02e-f951-4439-bb66-4bdf95cb1e97 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:13.919294 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2326b02e-f951-4439-bb66-4bdf95cb1e97 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:13.941833 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1012/2052] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:48:13 2022] GET /identity/v3/roles => generated 1427 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:48:13.948397 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff11522-cb5b-4af5-b7e3-97671f81c08a admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:13.950579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-3ff11522-cb5b-4af5-b7e3-97671f81c08a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:14.000815 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff11522-cb5b-4af5-b7e3-97671f81c08a admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:14.006687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58696e37-a90b-4032-95aa-6f3ff3a1297f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:14.008053 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58696e37-a90b-4032-95aa-6f3ff3a1297f admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:14.009091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58696e37-a90b-4032-95aa-6f3ff3a1297f admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:14.009545 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58696e37-a90b-4032-95aa-6f3ff3a1297f admin admin] PATH_INFO: `/v3/projects/6c4299ab3e584f579073d3bc2047c764/users/049e22dc9adb4579b3f3fe8af7865ca4/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:14.011611 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58696e37-a90b-4032-95aa-6f3ff3a1297f admin admin] RBAC: Authorizing `identity:create_grant(project_id=6c4299ab3e584f579073d3bc2047c764, user_id=049e22dc9adb4579b3f3fe8af7865ca4, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:14.017605 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58696e37-a90b-4032-95aa-6f3ff3a1297f admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:14.031005 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1041/2053] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:48:13 2022] PUT /identity/v3/projects/6c4299ab3e584f579073d3bc2047c764/users/049e22dc9adb4579b3f3fe8af7865ca4/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:14.049042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f835552b-8cc1-439c-82ec-24d4439b5d08 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:14.049348 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f835552b-8cc1-439c-82ec-24d4439b5d08 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:14.049604 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f835552b-8cc1-439c-82ec-24d4439b5d08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:14.090125 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-f835552b-8cc1-439c-82ec-24d4439b5d08 None None] MFA Rules not processed for user `049e22dc9adb4579b3f3fe8af7865ca4`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:14.091384 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-f835552b-8cc1-439c-82ec-24d4439b5d08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:14.156767 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:14.171758 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58696e37-a90b-4032-95aa-6f3ff3a1297f admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:14.173745 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-58696e37-a90b-4032-95aa-6f3ff3a1297f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:14.202123 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1013/2054] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:14 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:14.220279 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f835552b-8cc1-439c-82ec-24d4439b5d08 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:14.223311 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-f835552b-8cc1-439c-82ec-24d4439b5d08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:14.279841 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58696e37-a90b-4032-95aa-6f3ff3a1297f admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:14.285749 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2071eb67-a8c1-4b85-b859-f7ad264c7df6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:14.286514 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2071eb67-a8c1-4b85-b859-f7ad264c7df6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:14.286802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2071eb67-a8c1-4b85-b859-f7ad264c7df6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:14.287080 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2071eb67-a8c1-4b85-b859-f7ad264c7df6 admin admin] PATH_INFO: `/v3/projects/6558ebaf165c41bca42094829ed0c4ec` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:14.289280 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2071eb67-a8c1-4b85-b859-f7ad264c7df6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6558ebaf165c41bca42094829ed0c4ec)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:14.291519 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2071eb67-a8c1-4b85-b859-f7ad264c7df6 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:14.305459 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f835552b-8cc1-439c-82ec-24d4439b5d08 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:14.311107 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a3a6d1-31ec-4917-903e-1236fb8f1cbf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:14.312033 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a3a6d1-31ec-4917-903e-1236fb8f1cbf service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:14.312406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a3a6d1-31ec-4917-903e-1236fb8f1cbf service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:14.313342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a3a6d1-31ec-4917-903e-1236fb8f1cbf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:14.330228 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a3a6d1-31ec-4917-903e-1236fb8f1cbf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:14.332668 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a3a6d1-31ec-4917-903e-1236fb8f1cbf service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:14.345524 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1014/2055] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:48:14 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:48:14.356254 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-2071eb67-a8c1-4b85-b859-f7ad264c7df6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6558ebaf165c41bca42094829ed0c4ec', 'request_id': 'req-2071eb67-a8c1-4b85-b859-f7ad264c7df6'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:14.363442 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-2071eb67-a8c1-4b85-b859-f7ad264c7df6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6558ebaf165c41bca42094829ed0c4ec', 'request_id': 'req-2071eb67-a8c1-4b85-b859-f7ad264c7df6'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:14.365829 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:14.372074 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-2071eb67-a8c1-4b85-b859-f7ad264c7df6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6558ebaf165c41bca42094829ed0c4ec', 'request_id': 'req-2071eb67-a8c1-4b85-b859-f7ad264c7df6'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:14.389046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-2071eb67-a8c1-4b85-b859-f7ad264c7df6 admin admin] The token cache is being invalidate because project 6558ebaf165c41bca42094829ed0c4ec was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:48:14.389454 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-2071eb67-a8c1-4b85-b859-f7ad264c7df6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:14.390198 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-2071eb67-a8c1-4b85-b859-f7ad264c7df6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:14.391957 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1042/2056] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:48:14 2022] DELETE /identity/v3/projects/6558ebaf165c41bca42094829ed0c4ec => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:14.397109 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747c1b54-ffa4-4ead-9e24-0d3b0316da93 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:14.397379 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747c1b54-ffa4-4ead-9e24-0d3b0316da93 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:14.397379 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747c1b54-ffa4-4ead-9e24-0d3b0316da93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:14.420645 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-747c1b54-ffa4-4ead-9e24-0d3b0316da93 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:14.421927 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-747c1b54-ffa4-4ead-9e24-0d3b0316da93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:14.503278 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1015/2057] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:14 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:14.513155 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b98da4-5839-4263-9909-0cb06000adf0 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:14.513449 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b98da4-5839-4263-9909-0cb06000adf0 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:14.513732 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b98da4-5839-4263-9909-0cb06000adf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:14.536186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-93b98da4-5839-4263-9909-0cb06000adf0 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:14.537409 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-93b98da4-5839-4263-9909-0cb06000adf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:14.546378 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1043/2058] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:14 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:14.550870 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747c1b54-ffa4-4ead-9e24-0d3b0316da93 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:14.562109 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747c1b54-ffa4-4ead-9e24-0d3b0316da93 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:14.567491 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107908cd-2a72-42c2-9fb9-c3e5533e7cf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:14.568123 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107908cd-2a72-42c2-9fb9-c3e5533e7cf5 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:14.568409 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107908cd-2a72-42c2-9fb9-c3e5533e7cf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:14.568934 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107908cd-2a72-42c2-9fb9-c3e5533e7cf5 admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:14.569593 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-107908cd-2a72-42c2-9fb9-c3e5533e7cf5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:48:14.569896 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-107908cd-2a72-42c2-9fb9-c3e5533e7cf5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:14.572307 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-107908cd-2a72-42c2-9fb9-c3e5533e7cf5 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:14.578180 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b98da4-5839-4263-9909-0cb06000adf0 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:14.579365 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-93b98da4-5839-4263-9909-0cb06000adf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:14.587217 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1016/2059] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:48:14 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:48:14.591847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107908cd-2a72-42c2-9fb9-c3e5533e7cf5 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:14.604034 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107908cd-2a72-42c2-9fb9-c3e5533e7cf5 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:14.609859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f004b11e-813e-4ac6-9ca0-f381d980e4f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:14.610491 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f004b11e-813e-4ac6-9ca0-f381d980e4f9 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:14.610776 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f004b11e-813e-4ac6-9ca0-f381d980e4f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:14.611051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f004b11e-813e-4ac6-9ca0-f381d980e4f9 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:14.611795 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f004b11e-813e-4ac6-9ca0-f381d980e4f9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:14.613760 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f004b11e-813e-4ac6-9ca0-f381d980e4f9 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:14.646702 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b98da4-5839-4263-9909-0cb06000adf0 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:14.647311 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1017/2060] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:48:14 2022] POST /identity/v3/projects => generated 372 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 15:48:14.652194 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f004b11e-813e-4ac6-9ca0-f381d980e4f9 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:14.708775 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8c74b4-74fc-4011-b0be-1c442483247a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:14.709613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8c74b4-74fc-4011-b0be-1c442483247a service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:14.710132 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8c74b4-74fc-4011-b0be-1c442483247a service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:14.710387 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8c74b4-74fc-4011-b0be-1c442483247a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:14.711606 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:14.713654 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-4a8c74b4-74fc-4011-b0be-1c442483247a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:14.729004 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f004b11e-813e-4ac6-9ca0-f381d980e4f9 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:14.733964 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7386f1-3c7b-43c2-9b37-69939fa3e2e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:14.734599 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7386f1-3c7b-43c2-9b37-69939fa3e2e8 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:14.734948 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7386f1-3c7b-43c2-9b37-69939fa3e2e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:14.735290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7386f1-3c7b-43c2-9b37-69939fa3e2e8 admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:14.736032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b7386f1-3c7b-43c2-9b37-69939fa3e2e8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:14.738055 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b7386f1-3c7b-43c2-9b37-69939fa3e2e8 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:14.787419 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1018/2061] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:48:14 2022] POST /identity/v3/users => generated 287 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 15:48:14.792533 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7386f1-3c7b-43c2-9b37-69939fa3e2e8 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:14.808777 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7386f1-3c7b-43c2-9b37-69939fa3e2e8 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:14.816859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db33da83-99e4-4c39-b227-c56ca2cd2f2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:14.817679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db33da83-99e4-4c39-b227-c56ca2cd2f2c admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:14.817964 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db33da83-99e4-4c39-b227-c56ca2cd2f2c admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:14.818239 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db33da83-99e4-4c39-b227-c56ca2cd2f2c admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:14.819299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db33da83-99e4-4c39-b227-c56ca2cd2f2c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:14.821965 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db33da83-99e4-4c39-b227-c56ca2cd2f2c admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:14.826907 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8c74b4-74fc-4011-b0be-1c442483247a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:14.828769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8c74b4-74fc-4011-b0be-1c442483247a service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:14.843304 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1019/2062] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:48:14 2022] GET /identity/v3/roles => generated 1427 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:48:14.844635 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db33da83-99e4-4c39-b227-c56ca2cd2f2c admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:14.856551 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db33da83-99e4-4c39-b227-c56ca2cd2f2c admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:14.862472 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146c6de9-078e-4870-9535-feb759a34d0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:14.863088 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146c6de9-078e-4870-9535-feb759a34d0d admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:14.863395 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146c6de9-078e-4870-9535-feb759a34d0d admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:14.863742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146c6de9-078e-4870-9535-feb759a34d0d admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:14.864609 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-146c6de9-078e-4870-9535-feb759a34d0d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:14.866354 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-146c6de9-078e-4870-9535-feb759a34d0d admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:14.875330 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1044/2063] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:48:14 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 286 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:48:14.883721 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1020/2064] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:48:14 2022] GET /identity/v3/roles => generated 1427 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:48:14.885198 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8c74b4-74fc-4011-b0be-1c442483247a service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:14.921011 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8c74b4-74fc-4011-b0be-1c442483247a service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:14.926771 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed50c8a4-c4ff-4423-8c50-615c5f10ce05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:14.927517 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed50c8a4-c4ff-4423-8c50-615c5f10ce05 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:14.927940 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed50c8a4-c4ff-4423-8c50-615c5f10ce05 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:14.928327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed50c8a4-c4ff-4423-8c50-615c5f10ce05 admin admin] PATH_INFO: `/v3/projects/5f10c9af056e49c1b8dec58e6c3f8710/users/77d26750fc8c4863b1a2b73e78edc0db/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:14.961886 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed50c8a4-c4ff-4423-8c50-615c5f10ce05 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5f10c9af056e49c1b8dec58e6c3f8710, user_id=77d26750fc8c4863b1a2b73e78edc0db, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:14.965077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed50c8a4-c4ff-4423-8c50-615c5f10ce05 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:15.015329 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1045/2065] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:48:14 2022] PUT /identity/v3/projects/5f10c9af056e49c1b8dec58e6c3f8710/users/77d26750fc8c4863b1a2b73e78edc0db/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:15.027557 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f767f5d-251f-4567-b9e5-b038559b316e None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:15.027902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f767f5d-251f-4567-b9e5-b038559b316e None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:15.028135 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f767f5d-251f-4567-b9e5-b038559b316e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:15.089064 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-6f767f5d-251f-4567-b9e5-b038559b316e None None] MFA Rules not processed for user `77d26750fc8c4863b1a2b73e78edc0db`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:15.090679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-6f767f5d-251f-4567-b9e5-b038559b316e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:15.209628 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1021/2066] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:15 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:15.214486 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed50c8a4-c4ff-4423-8c50-615c5f10ce05 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:15.254781 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=96,keystone:DELETE=2,keystone:INSERT=2 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:15.303620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed50c8a4-c4ff-4423-8c50-615c5f10ce05 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:15.309446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1289a2-dc5a-4499-829a-86b532434042 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:15.310026 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1289a2-dc5a-4499-829a-86b532434042 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:15.310315 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1289a2-dc5a-4499-829a-86b532434042 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:15.310623 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1289a2-dc5a-4499-829a-86b532434042 admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:15.311605 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea1289a2-dc5a-4499-829a-86b532434042 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:15.313419 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea1289a2-dc5a-4499-829a-86b532434042 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:15.346631 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f767f5d-251f-4567-b9e5-b038559b316e None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:15.348420 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-6f767f5d-251f-4567-b9e5-b038559b316e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:15.362339 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1046/2067] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:48:15 2022] POST /identity/v3/projects => generated 376 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:48:15.363270 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1289a2-dc5a-4499-829a-86b532434042 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:15.369790 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f767f5d-251f-4567-b9e5-b038559b316e None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:15.463277 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c7a0a8-1e58-49dc-92f4-d1537297c594 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:15.464305 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c7a0a8-1e58-49dc-92f4-d1537297c594 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:15.464745 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c7a0a8-1e58-49dc-92f4-d1537297c594 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:15.465144 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c7a0a8-1e58-49dc-92f4-d1537297c594 admin admin] PATH_INFO: `/v3/users/c18a557fc5484e85b6a3353ad944c133` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:15.466347 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c7a0a8-1e58-49dc-92f4-d1537297c594 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c18a557fc5484e85b6a3353ad944c133)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:15.468634 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c7a0a8-1e58-49dc-92f4-d1537297c594 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:15.483815 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1289a2-dc5a-4499-829a-86b532434042 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:15.491445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb9fc49-b16b-4fb8-aea6-0fdabbfa9e26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:15.492761 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb9fc49-b16b-4fb8-aea6-0fdabbfa9e26 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:15.493075 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb9fc49-b16b-4fb8-aea6-0fdabbfa9e26 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:15.493375 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb9fc49-b16b-4fb8-aea6-0fdabbfa9e26 admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:15.494127 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb9fc49-b16b-4fb8-aea6-0fdabbfa9e26 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:15.497585 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb9fc49-b16b-4fb8-aea6-0fdabbfa9e26 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:15.521508 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1047/2068] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:48:15 2022] POST /identity/v3/users => generated 295 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:48:15.527248 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb9fc49-b16b-4fb8-aea6-0fdabbfa9e26 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:15.540563 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb9fc49-b16b-4fb8-aea6-0fdabbfa9e26 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:15.548919 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40763b2c-98eb-4121-89b6-afe017ef7f5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:15.549619 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40763b2c-98eb-4121-89b6-afe017ef7f5c admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:15.550095 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40763b2c-98eb-4121-89b6-afe017ef7f5c admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:15.551370 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40763b2c-98eb-4121-89b6-afe017ef7f5c admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:15.552189 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40763b2c-98eb-4121-89b6-afe017ef7f5c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:15.555420 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40763b2c-98eb-4121-89b6-afe017ef7f5c admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:15.569509 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1048/2069] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:48:15 2022] GET /identity/v3/roles => generated 1427 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:48:15.573496 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-b1c7a0a8-1e58-49dc-92f4-d1537297c594 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c18a557fc5484e85b6a3353ad944c133'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:15.574268 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-b1c7a0a8-1e58-49dc-92f4-d1537297c594 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c18a557fc5484e85b6a3353ad944c133'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:15.574946 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40763b2c-98eb-4121-89b6-afe017ef7f5c admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:15.580669 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:15.590070 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-b1c7a0a8-1e58-49dc-92f4-d1537297c594 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c18a557fc5484e85b6a3353ad944c133'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:15.598304 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40763b2c-98eb-4121-89b6-afe017ef7f5c admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:15.608570 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-b1c7a0a8-1e58-49dc-92f4-d1537297c594 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c18a557fc5484e85b6a3353ad944c133'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:15.609155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-b1c7a0a8-1e58-49dc-92f4-d1537297c594 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c18a557fc5484e85b6a3353ad944c133'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:15.616650 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc120ff8-7952-4c68-a927-a0f6646ab10d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:15.617488 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc120ff8-7952-4c68-a927-a0f6646ab10d admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:15.618657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc120ff8-7952-4c68-a927-a0f6646ab10d admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:15.618980 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc120ff8-7952-4c68-a927-a0f6646ab10d admin admin] PATH_INFO: `/v3/projects/b7a3a225cecf4dc0976b9b0f9f74ba87/users/a3f99dc0e8ae49f4b067eeb042e28717/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:15.631572 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1022/2070] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:48:15 2022] DELETE /identity/v3/users/c18a557fc5484e85b6a3353ad944c133 => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:15.636419 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afa71b8-1b4a-4573-afca-ce3d6f1b5346 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:15.636669 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afa71b8-1b4a-4573-afca-ce3d6f1b5346 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:15.636948 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afa71b8-1b4a-4573-afca-ce3d6f1b5346 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:15.637427 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc120ff8-7952-4c68-a927-a0f6646ab10d admin admin] RBAC: Authorizing `identity:create_grant(project_id=b7a3a225cecf4dc0976b9b0f9f74ba87, user_id=a3f99dc0e8ae49f4b067eeb042e28717, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:15.640268 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc120ff8-7952-4c68-a927-a0f6646ab10d admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:15.657598 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1049/2071] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:48:15 2022] PUT /identity/v3/projects/b7a3a225cecf4dc0976b9b0f9f74ba87/users/a3f99dc0e8ae49f4b067eeb042e28717/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 15:48:15.657803 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-5afa71b8-1b4a-4573-afca-ce3d6f1b5346 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:15.659099 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-5afa71b8-1b4a-4573-afca-ce3d6f1b5346 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:15.664890 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc120ff8-7952-4c68-a927-a0f6646ab10d admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:15.667808 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-cc120ff8-7952-4c68-a927-a0f6646ab10d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:15.740362 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1023/2072] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:15 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:48:15.751396 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc120ff8-7952-4c68-a927-a0f6646ab10d admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:15.754395 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5afa71b8-1b4a-4573-afca-ce3d6f1b5346 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:15.755905 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-5afa71b8-1b4a-4573-afca-ce3d6f1b5346 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:15.758513 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ddbc51c-1408-4b1d-a0d5-8634daac0a6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:15.759228 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddbc51c-1408-4b1d-a0d5-8634daac0a6f admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:15.759580 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddbc51c-1408-4b1d-a0d5-8634daac0a6f admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:15.759951 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddbc51c-1408-4b1d-a0d5-8634daac0a6f admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:15.760689 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ddbc51c-1408-4b1d-a0d5-8634daac0a6f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:15.762358 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ddbc51c-1408-4b1d-a0d5-8634daac0a6f admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:15.769453 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:15.775824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1050/2073] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:48:15 2022] GET /identity/v3/roles => generated 1427 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:48:15.780628 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ddbc51c-1408-4b1d-a0d5-8634daac0a6f admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:15.793297 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ddbc51c-1408-4b1d-a0d5-8634daac0a6f admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:15.799865 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98df28a-612d-41b5-9a50-ccdb5d3aa46d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:15.800687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98df28a-612d-41b5-9a50-ccdb5d3aa46d admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:15.801070 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98df28a-612d-41b5-9a50-ccdb5d3aa46d admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:15.801467 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98df28a-612d-41b5-9a50-ccdb5d3aa46d admin admin] PATH_INFO: `/v3/projects/b7a3a225cecf4dc0976b9b0f9f74ba87/users/a3f99dc0e8ae49f4b067eeb042e28717/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:15.803701 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b98df28a-612d-41b5-9a50-ccdb5d3aa46d admin admin] RBAC: Authorizing `identity:create_grant(project_id=b7a3a225cecf4dc0976b9b0f9f74ba87, user_id=a3f99dc0e8ae49f4b067eeb042e28717, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:15.807433 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b98df28a-612d-41b5-9a50-ccdb5d3aa46d admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:15.819946 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1051/2074] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:48:15 2022] PUT /identity/v3/projects/b7a3a225cecf4dc0976b9b0f9f74ba87/users/a3f99dc0e8ae49f4b067eeb042e28717/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 15:48:15.830983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fad89b1-e16a-4524-8546-8e8093ea3656 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:15.831261 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fad89b1-e16a-4524-8546-8e8093ea3656 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:15.831623 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fad89b1-e16a-4524-8546-8e8093ea3656 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:15.840563 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5afa71b8-1b4a-4573-afca-ce3d6f1b5346 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:15.863432 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-1fad89b1-e16a-4524-8546-8e8093ea3656 None None] MFA Rules not processed for user `a3f99dc0e8ae49f4b067eeb042e28717`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:15.864901 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-1fad89b1-e16a-4524-8546-8e8093ea3656 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:15.909439 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f278f366-5e98-4ce6-949f-ca7281474f6a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:15.910218 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f278f366-5e98-4ce6-949f-ca7281474f6a service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:15.910505 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f278f366-5e98-4ce6-949f-ca7281474f6a service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:15.910833 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f278f366-5e98-4ce6-949f-ca7281474f6a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:15.978126 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=96,keystone:DELETE=3,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:15.981834 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1052/2075] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:15 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:15.983329 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fad89b1-e16a-4524-8546-8e8093ea3656 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:15.990351 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-1fad89b1-e16a-4524-8546-8e8093ea3656 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:16.009946 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f278f366-5e98-4ce6-949f-ca7281474f6a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:16.011567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f278f366-5e98-4ce6-949f-ca7281474f6a service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:16.024331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1024/2076] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:48:15 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 270 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 15:48:16.084733 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fad89b1-e16a-4524-8546-8e8093ea3656 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:16.090042 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb72a0a3-a319-4e42-8c0b-08819772f828 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:16.090663 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb72a0a3-a319-4e42-8c0b-08819772f828 service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:16.090943 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb72a0a3-a319-4e42-8c0b-08819772f828 service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:16.091213 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb72a0a3-a319-4e42-8c0b-08819772f828 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:16.093137 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-fb72a0a3-a319-4e42-8c0b-08819772f828 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:16.201403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb72a0a3-a319-4e42-8c0b-08819772f828 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:16.202855 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb72a0a3-a319-4e42-8c0b-08819772f828 service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:16.213482 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1053/2077] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:48:15 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 231 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:48:16.593901 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f278f366-5e98-4ce6-949f-ca7281474f6a service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:16.595368 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-f278f366-5e98-4ce6-949f-ca7281474f6a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:16.612015 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f278f366-5e98-4ce6-949f-ca7281474f6a service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:16.620257 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd44b10-8e54-4e0b-9dfb-39b7776c685f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:16.621026 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd44b10-8e54-4e0b-9dfb-39b7776c685f admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:16.621372 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd44b10-8e54-4e0b-9dfb-39b7776c685f admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:16.621720 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd44b10-8e54-4e0b-9dfb-39b7776c685f admin admin] PATH_INFO: `/v3/projects/5b7bc49fffc74a2ba1c77dbc24710d5d` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:16.623052 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd44b10-8e54-4e0b-9dfb-39b7776c685f admin admin] RBAC: Authorizing `identity:delete_project(project_id=5b7bc49fffc74a2ba1c77dbc24710d5d)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:16.625404 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd44b10-8e54-4e0b-9dfb-39b7776c685f admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:16.685973 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-dbd44b10-8e54-4e0b-9dfb-39b7776c685f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5b7bc49fffc74a2ba1c77dbc24710d5d', 'request_id': 'req-dbd44b10-8e54-4e0b-9dfb-39b7776c685f'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:16.694144 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-dbd44b10-8e54-4e0b-9dfb-39b7776c685f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5b7bc49fffc74a2ba1c77dbc24710d5d', 'request_id': 'req-dbd44b10-8e54-4e0b-9dfb-39b7776c685f'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:16.712993 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-dbd44b10-8e54-4e0b-9dfb-39b7776c685f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5b7bc49fffc74a2ba1c77dbc24710d5d', 'request_id': 'req-dbd44b10-8e54-4e0b-9dfb-39b7776c685f'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:16.731173 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-dbd44b10-8e54-4e0b-9dfb-39b7776c685f admin admin] The token cache is being invalidate because project 5b7bc49fffc74a2ba1c77dbc24710d5d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:48:16.731726 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-dbd44b10-8e54-4e0b-9dfb-39b7776c685f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:16.732823 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-dbd44b10-8e54-4e0b-9dfb-39b7776c685f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:16.737404 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1025/2078] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:48:16 2022] DELETE /identity/v3/projects/5b7bc49fffc74a2ba1c77dbc24710d5d => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:16.739013 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb72a0a3-a319-4e42-8c0b-08819772f828 service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:16.754465 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb72a0a3-a319-4e42-8c0b-08819772f828 service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:16.783051 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:16.822925 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a50caea-4efd-46f1-8624-5084b736983e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:16.823432 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a50caea-4efd-46f1-8624-5084b736983e admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:16.823744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a50caea-4efd-46f1-8624-5084b736983e admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:16.824025 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a50caea-4efd-46f1-8624-5084b736983e admin admin] PATH_INFO: `/v3/users/049e22dc9adb4579b3f3fe8af7865ca4` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:16.825194 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a50caea-4efd-46f1-8624-5084b736983e admin admin] RBAC: Authorizing `identity:delete_user(user_id=049e22dc9adb4579b3f3fe8af7865ca4)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:16.827151 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a50caea-4efd-46f1-8624-5084b736983e admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:16.885775 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-6a50caea-4efd-46f1-8624-5084b736983e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '049e22dc9adb4579b3f3fe8af7865ca4'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:16.886612 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-6a50caea-4efd-46f1-8624-5084b736983e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '049e22dc9adb4579b3f3fe8af7865ca4'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:16.887230 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-6a50caea-4efd-46f1-8624-5084b736983e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '049e22dc9adb4579b3f3fe8af7865ca4'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:16.894641 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-6a50caea-4efd-46f1-8624-5084b736983e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '049e22dc9adb4579b3f3fe8af7865ca4'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:16.907234 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-6a50caea-4efd-46f1-8624-5084b736983e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '049e22dc9adb4579b3f3fe8af7865ca4'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:16.921694 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1054/2079] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:48:16 2022] DELETE /identity/v3/users/049e22dc9adb4579b3f3fe8af7865ca4 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:17.216503 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd44b10-8e54-4e0b-9dfb-39b7776c685f admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:17.218314 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-dbd44b10-8e54-4e0b-9dfb-39b7776c685f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:17.239365 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd44b10-8e54-4e0b-9dfb-39b7776c685f admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:17.302994 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff97be5-41aa-4c63-9b07-5aa0e56dff0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:17.303639 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff97be5-41aa-4c63-9b07-5aa0e56dff0f admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:17.303976 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff97be5-41aa-4c63-9b07-5aa0e56dff0f admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:17.304224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff97be5-41aa-4c63-9b07-5aa0e56dff0f admin admin] PATH_INFO: `/v3/projects/6c4299ab3e584f579073d3bc2047c764` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:17.305409 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff97be5-41aa-4c63-9b07-5aa0e56dff0f admin admin] RBAC: Authorizing `identity:delete_project(project_id=6c4299ab3e584f579073d3bc2047c764)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:17.307365 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff97be5-41aa-4c63-9b07-5aa0e56dff0f admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:17.387112 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-2ff97be5-41aa-4c63-9b07-5aa0e56dff0f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6c4299ab3e584f579073d3bc2047c764', 'request_id': 'req-2ff97be5-41aa-4c63-9b07-5aa0e56dff0f'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:17.395327 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-2ff97be5-41aa-4c63-9b07-5aa0e56dff0f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6c4299ab3e584f579073d3bc2047c764', 'request_id': 'req-2ff97be5-41aa-4c63-9b07-5aa0e56dff0f'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:17.401826 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:17.402781 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-2ff97be5-41aa-4c63-9b07-5aa0e56dff0f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6c4299ab3e584f579073d3bc2047c764', 'request_id': 'req-2ff97be5-41aa-4c63-9b07-5aa0e56dff0f'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:17.419456 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-2ff97be5-41aa-4c63-9b07-5aa0e56dff0f admin admin] The token cache is being invalidate because project 6c4299ab3e584f579073d3bc2047c764 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:48:17.420019 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-2ff97be5-41aa-4c63-9b07-5aa0e56dff0f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:17.421163 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-2ff97be5-41aa-4c63-9b07-5aa0e56dff0f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:17.423244 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1026/2080] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:48:17 2022] DELETE /identity/v3/projects/6c4299ab3e584f579073d3bc2047c764 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:17.429504 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df72e94-71f6-499d-993e-4d8bc7dd490e None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:17.432422 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df72e94-71f6-499d-993e-4d8bc7dd490e None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:17.432762 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df72e94-71f6-499d-993e-4d8bc7dd490e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:17.458445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-2df72e94-71f6-499d-993e-4d8bc7dd490e None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:17.459739 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-2df72e94-71f6-499d-993e-4d8bc7dd490e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:17.534891 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1055/2081] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:17 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:17.543606 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5291c1-5847-4d52-afbd-07d6fd048b01 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:17.543927 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5291c1-5847-4d52-afbd-07d6fd048b01 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:17.544211 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5291c1-5847-4d52-afbd-07d6fd048b01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:17.564156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-bd5291c1-5847-4d52-afbd-07d6fd048b01 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:17.565378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-bd5291c1-5847-4d52-afbd-07d6fd048b01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:17.579017 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1027/2082] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:17 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:17.584704 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df72e94-71f6-499d-993e-4d8bc7dd490e None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:17.597077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df72e94-71f6-499d-993e-4d8bc7dd490e None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:17.604428 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5eddd0-6d31-4f2f-83ec-bd0a579c57fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:17.605029 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5eddd0-6d31-4f2f-83ec-bd0a579c57fd admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:17.605350 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5eddd0-6d31-4f2f-83ec-bd0a579c57fd admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:17.605673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5eddd0-6d31-4f2f-83ec-bd0a579c57fd admin admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:17.606331 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb5eddd0-6d31-4f2f-83ec-bd0a579c57fd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:48:17.606685 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb5eddd0-6d31-4f2f-83ec-bd0a579c57fd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:17.608344 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb5eddd0-6d31-4f2f-83ec-bd0a579c57fd admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:17.627498 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1056/2083] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:48:17 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:48:17.632210 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5291c1-5847-4d52-afbd-07d6fd048b01 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:17.646754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5291c1-5847-4d52-afbd-07d6fd048b01 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:17.653744 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabd76aa-45be-4ce7-96c0-8c6d3be09187 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:17.655081 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabd76aa-45be-4ce7-96c0-8c6d3be09187 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:17.655429 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabd76aa-45be-4ce7-96c0-8c6d3be09187 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:17.655813 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabd76aa-45be-4ce7-96c0-8c6d3be09187 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:17.656603 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dabd76aa-45be-4ce7-96c0-8c6d3be09187 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:17.658703 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dabd76aa-45be-4ce7-96c0-8c6d3be09187 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:17.692472 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1028/2084] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:48:17 2022] POST /identity/v3/projects => generated 384 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:48:17.694428 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5eddd0-6d31-4f2f-83ec-bd0a579c57fd admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:17.706802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5eddd0-6d31-4f2f-83ec-bd0a579c57fd admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:17.743949 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:17.782466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8144c01-32b3-41bd-9c1e-e557d90d4870 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:17.783101 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8144c01-32b3-41bd-9c1e-e557d90d4870 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:17.783392 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8144c01-32b3-41bd-9c1e-e557d90d4870 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:17.785165 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8144c01-32b3-41bd-9c1e-e557d90d4870 admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:17.785991 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8144c01-32b3-41bd-9c1e-e557d90d4870 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:17.788773 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8144c01-32b3-41bd-9c1e-e557d90d4870 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:17.829010 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1057/2085] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:48:17 2022] POST /identity/v3/users => generated 293 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 15:48:17.832919 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabd76aa-45be-4ce7-96c0-8c6d3be09187 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:17.856181 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabd76aa-45be-4ce7-96c0-8c6d3be09187 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:17.867766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b6ed98-d506-43ad-89e2-0c0f88e4e703 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:17.870539 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b6ed98-d506-43ad-89e2-0c0f88e4e703 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:17.870833 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b6ed98-d506-43ad-89e2-0c0f88e4e703 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:17.871111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b6ed98-d506-43ad-89e2-0c0f88e4e703 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:17.871830 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58b6ed98-d506-43ad-89e2-0c0f88e4e703 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:17.873457 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58b6ed98-d506-43ad-89e2-0c0f88e4e703 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:17.896488 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8144c01-32b3-41bd-9c1e-e557d90d4870 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:17.903843 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1029/2086] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:48:17 2022] GET /identity/v3/roles => generated 1427 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:48:17.914890 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8144c01-32b3-41bd-9c1e-e557d90d4870 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:17.920709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479e589f-0e34-4568-910c-5ba75b0f7d21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:17.923331 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479e589f-0e34-4568-910c-5ba75b0f7d21 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:17.923331 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479e589f-0e34-4568-910c-5ba75b0f7d21 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:17.923331 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479e589f-0e34-4568-910c-5ba75b0f7d21 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:17.923331 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479e589f-0e34-4568-910c-5ba75b0f7d21 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:17.924416 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479e589f-0e34-4568-910c-5ba75b0f7d21 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:17.951672 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1058/2087] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:48:17 2022] GET /identity/v3/roles => generated 1427 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:48:17.954170 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b6ed98-d506-43ad-89e2-0c0f88e4e703 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:17.966892 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b6ed98-d506-43ad-89e2-0c0f88e4e703 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:17.975012 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8af213-ecd5-4fb2-9f6e-132a270deb2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:17.975713 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8af213-ecd5-4fb2-9f6e-132a270deb2f admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:17.975995 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8af213-ecd5-4fb2-9f6e-132a270deb2f admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:17.976349 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8af213-ecd5-4fb2-9f6e-132a270deb2f admin admin] PATH_INFO: `/v3/projects/eb6a9b296ce74b569d52df41c8449820/users/4248ebe20e934d78800726fb766d7086/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:17.995359 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c8af213-ecd5-4fb2-9f6e-132a270deb2f admin admin] RBAC: Authorizing `identity:create_grant(project_id=eb6a9b296ce74b569d52df41c8449820, user_id=4248ebe20e934d78800726fb766d7086, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:17.998152 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c8af213-ecd5-4fb2-9f6e-132a270deb2f admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:18.011984 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1030/2088] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:48:17 2022] PUT /identity/v3/projects/eb6a9b296ce74b569d52df41c8449820/users/4248ebe20e934d78800726fb766d7086/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:18.022376 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807bf552-b103-44fd-9a54-3dde43528c10 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:18.022661 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807bf552-b103-44fd-9a54-3dde43528c10 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:18.022942 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807bf552-b103-44fd-9a54-3dde43528c10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:18.053037 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-807bf552-b103-44fd-9a54-3dde43528c10 None None] MFA Rules not processed for user `4248ebe20e934d78800726fb766d7086`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:18.054811 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-807bf552-b103-44fd-9a54-3dde43528c10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:18.134195 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1059/2089] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:18 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:18.145985 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8af213-ecd5-4fb2-9f6e-132a270deb2f admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:18.147861 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-3c8af213-ecd5-4fb2-9f6e-132a270deb2f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:18.235948 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8af213-ecd5-4fb2-9f6e-132a270deb2f admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:18.241722 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50701248-8909-4ef5-8935-b028e97e1e8f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:18.241722 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50701248-8909-4ef5-8935-b028e97e1e8f service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:18.241722 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50701248-8909-4ef5-8935-b028e97e1e8f service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:18.242243 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50701248-8909-4ef5-8935-b028e97e1e8f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:18.252379 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50701248-8909-4ef5-8935-b028e97e1e8f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:18.254757 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50701248-8909-4ef5-8935-b028e97e1e8f service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:18.265985 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1031/2090] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:48:18 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:48:18.969074 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807bf552-b103-44fd-9a54-3dde43528c10 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:18.970474 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-807bf552-b103-44fd-9a54-3dde43528c10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:19.045598 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807bf552-b103-44fd-9a54-3dde43528c10 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:19.050758 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c119b4df-1838-4099-9799-cf3f2f00d85c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:19.051471 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c119b4df-1838-4099-9799-cf3f2f00d85c admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:19.051864 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c119b4df-1838-4099-9799-cf3f2f00d85c admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:19.052538 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c119b4df-1838-4099-9799-cf3f2f00d85c admin admin] PATH_INFO: `/v3/users/77d26750fc8c4863b1a2b73e78edc0db` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:19.056347 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c119b4df-1838-4099-9799-cf3f2f00d85c admin admin] RBAC: Authorizing `identity:delete_user(user_id=77d26750fc8c4863b1a2b73e78edc0db)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:19.058345 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c119b4df-1838-4099-9799-cf3f2f00d85c admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:19.112003 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=91,keystone:INSERT=3,keystone:DELETE=6 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:19.114863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-c119b4df-1838-4099-9799-cf3f2f00d85c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '77d26750fc8c4863b1a2b73e78edc0db'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:19.115849 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-c119b4df-1838-4099-9799-cf3f2f00d85c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '77d26750fc8c4863b1a2b73e78edc0db'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:19.117262 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-c119b4df-1838-4099-9799-cf3f2f00d85c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '77d26750fc8c4863b1a2b73e78edc0db'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:19.126402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-c119b4df-1838-4099-9799-cf3f2f00d85c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '77d26750fc8c4863b1a2b73e78edc0db'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:19.143900 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-c119b4df-1838-4099-9799-cf3f2f00d85c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '77d26750fc8c4863b1a2b73e78edc0db'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:19.157624 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1060/2091] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:48:18 2022] DELETE /identity/v3/users/77d26750fc8c4863b1a2b73e78edc0db => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:19.161982 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e086237-b2a7-48c8-84ca-79b1cc1f6535 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:19.162600 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e086237-b2a7-48c8-84ca-79b1cc1f6535 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:19.162600 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e086237-b2a7-48c8-84ca-79b1cc1f6535 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:19.185824 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-0e086237-b2a7-48c8-84ca-79b1cc1f6535 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:19.187134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-0e086237-b2a7-48c8-84ca-79b1cc1f6535 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:19.253590 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1032/2092] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:19 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:19.264014 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c119b4df-1838-4099-9799-cf3f2f00d85c admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:19.265339 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-c119b4df-1838-4099-9799-cf3f2f00d85c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:19.335192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c119b4df-1838-4099-9799-cf3f2f00d85c admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:19.340157 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27179ff-b0a8-4dba-aafa-058796a77022 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:19.340871 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27179ff-b0a8-4dba-aafa-058796a77022 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:19.341151 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27179ff-b0a8-4dba-aafa-058796a77022 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:19.341428 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27179ff-b0a8-4dba-aafa-058796a77022 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:19.353576 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27179ff-b0a8-4dba-aafa-058796a77022 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:19.355014 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27179ff-b0a8-4dba-aafa-058796a77022 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:19.365259 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1061/2093] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:48:19 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:48:19.734929 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e086237-b2a7-48c8-84ca-79b1cc1f6535 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:19.736582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-0e086237-b2a7-48c8-84ca-79b1cc1f6535 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:19.748575 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e086237-b2a7-48c8-84ca-79b1cc1f6535 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:19.753329 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78689564-2699-447e-aa9e-490016c72cd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:19.759918 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78689564-2699-447e-aa9e-490016c72cd7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:19.760241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78689564-2699-447e-aa9e-490016c72cd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:19.760514 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78689564-2699-447e-aa9e-490016c72cd7 admin admin] PATH_INFO: `/v3/projects/5f10c9af056e49c1b8dec58e6c3f8710` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:19.761796 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78689564-2699-447e-aa9e-490016c72cd7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5f10c9af056e49c1b8dec58e6c3f8710)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:19.768733 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78689564-2699-447e-aa9e-490016c72cd7 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:19.797470 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=95,keystone:INSERT=3,keystone:DELETE=2 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:19.834156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-78689564-2699-447e-aa9e-490016c72cd7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5f10c9af056e49c1b8dec58e6c3f8710', 'request_id': 'req-78689564-2699-447e-aa9e-490016c72cd7'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:19.843828 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-78689564-2699-447e-aa9e-490016c72cd7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5f10c9af056e49c1b8dec58e6c3f8710', 'request_id': 'req-78689564-2699-447e-aa9e-490016c72cd7'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:19.851877 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-78689564-2699-447e-aa9e-490016c72cd7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5f10c9af056e49c1b8dec58e6c3f8710', 'request_id': 'req-78689564-2699-447e-aa9e-490016c72cd7'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:19.867857 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-78689564-2699-447e-aa9e-490016c72cd7 admin admin] The token cache is being invalidate because project 5f10c9af056e49c1b8dec58e6c3f8710 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:48:19.868230 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-78689564-2699-447e-aa9e-490016c72cd7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:19.868831 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-78689564-2699-447e-aa9e-490016c72cd7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:19.870690 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1033/2094] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:48:19 2022] DELETE /identity/v3/projects/5f10c9af056e49c1b8dec58e6c3f8710 => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:19.875533 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27179ff-b0a8-4dba-aafa-058796a77022 service neutron] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:19.888069 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27179ff-b0a8-4dba-aafa-058796a77022 service neutron] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:19.968055 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7e884a-faad-4267-a551-1b0edfb70271 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:19.968690 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7e884a-faad-4267-a551-1b0edfb70271 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:19.968968 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7e884a-faad-4267-a551-1b0edfb70271 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:19.969238 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7e884a-faad-4267-a551-1b0edfb70271 admin admin] PATH_INFO: `/v3/users/a3f99dc0e8ae49f4b067eeb042e28717` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:19.970574 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7e884a-faad-4267-a551-1b0edfb70271 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a3f99dc0e8ae49f4b067eeb042e28717)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:19.972638 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7e884a-faad-4267-a551-1b0edfb70271 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:20.025661 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-ce7e884a-faad-4267-a551-1b0edfb70271 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a3f99dc0e8ae49f4b067eeb042e28717'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:20.026387 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-ce7e884a-faad-4267-a551-1b0edfb70271 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a3f99dc0e8ae49f4b067eeb042e28717'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:20.027061 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-ce7e884a-faad-4267-a551-1b0edfb70271 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a3f99dc0e8ae49f4b067eeb042e28717'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:20.034176 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-ce7e884a-faad-4267-a551-1b0edfb70271 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a3f99dc0e8ae49f4b067eeb042e28717'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:20.042732 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:DELETE=13,keystone:SELECT=85,keystone:INSERT=2 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:20.049010 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-ce7e884a-faad-4267-a551-1b0edfb70271 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a3f99dc0e8ae49f4b067eeb042e28717'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:20.063154 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1062/2095] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:48:19 2022] DELETE /identity/v3/users/a3f99dc0e8ae49f4b067eeb042e28717 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:20.447316 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78689564-2699-447e-aa9e-490016c72cd7 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:20.449148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-78689564-2699-447e-aa9e-490016c72cd7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:20.466458 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78689564-2699-447e-aa9e-490016c72cd7 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:20.567252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d5c910-ba7d-479e-831e-37c48d4f83f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:20.568416 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d5c910-ba7d-479e-831e-37c48d4f83f4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:20.568416 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d5c910-ba7d-479e-831e-37c48d4f83f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:20.568653 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d5c910-ba7d-479e-831e-37c48d4f83f4 admin admin] PATH_INFO: `/v3/projects/b7a3a225cecf4dc0976b9b0f9f74ba87` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:20.569945 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d5c910-ba7d-479e-831e-37c48d4f83f4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b7a3a225cecf4dc0976b9b0f9f74ba87)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:20.572596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d5c910-ba7d-479e-831e-37c48d4f83f4 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:20.624961 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-78d5c910-ba7d-479e-831e-37c48d4f83f4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b7a3a225cecf4dc0976b9b0f9f74ba87', 'request_id': 'req-78d5c910-ba7d-479e-831e-37c48d4f83f4'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:20.633062 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-78d5c910-ba7d-479e-831e-37c48d4f83f4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b7a3a225cecf4dc0976b9b0f9f74ba87', 'request_id': 'req-78d5c910-ba7d-479e-831e-37c48d4f83f4'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:20.641049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-78d5c910-ba7d-479e-831e-37c48d4f83f4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b7a3a225cecf4dc0976b9b0f9f74ba87', 'request_id': 'req-78d5c910-ba7d-479e-831e-37c48d4f83f4'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:20.657504 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-78d5c910-ba7d-479e-831e-37c48d4f83f4 admin admin] The token cache is being invalidate because project b7a3a225cecf4dc0976b9b0f9f74ba87 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:48:20.657806 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-78d5c910-ba7d-479e-831e-37c48d4f83f4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:20.658897 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-78d5c910-ba7d-479e-831e-37c48d4f83f4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:20.660617 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1034/2096] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:48:20 2022] DELETE /identity/v3/projects/b7a3a225cecf4dc0976b9b0f9f74ba87 => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:20.665926 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a128f84f-9e06-4c94-8697-93b540559dd9 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:20.666244 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a128f84f-9e06-4c94-8697-93b540559dd9 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:20.666604 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a128f84f-9e06-4c94-8697-93b540559dd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:20.690130 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-a128f84f-9e06-4c94-8697-93b540559dd9 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:20.691728 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-a128f84f-9e06-4c94-8697-93b540559dd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:20.773817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1063/2097] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:20 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:20.783461 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaba7bab-0bf0-4274-9776-4854e9ed3509 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:20.783825 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaba7bab-0bf0-4274-9776-4854e9ed3509 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:20.784304 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaba7bab-0bf0-4274-9776-4854e9ed3509 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:20.816508 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-eaba7bab-0bf0-4274-9776-4854e9ed3509 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:20.817980 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-eaba7bab-0bf0-4274-9776-4854e9ed3509 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:20.828090 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1035/2098] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:20 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:20.832730 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a128f84f-9e06-4c94-8697-93b540559dd9 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:20.866145 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a128f84f-9e06-4c94-8697-93b540559dd9 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:20.871859 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e90d44-ff8e-427a-ab6a-94231f42e9be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:20.872480 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e90d44-ff8e-427a-ab6a-94231f42e9be admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:20.872758 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e90d44-ff8e-427a-ab6a-94231f42e9be admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:20.873029 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e90d44-ff8e-427a-ab6a-94231f42e9be admin admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:20.874699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e90d44-ff8e-427a-ab6a-94231f42e9be admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:48:20.875187 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e90d44-ff8e-427a-ab6a-94231f42e9be admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:20.877840 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e90d44-ff8e-427a-ab6a-94231f42e9be admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:20.895511 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1064/2099] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:48:20 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:48:20.896427 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaba7bab-0bf0-4274-9776-4854e9ed3509 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:20.908900 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaba7bab-0bf0-4274-9776-4854e9ed3509 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:20.928515 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534669e5-f927-44f4-9de8-9647f692f475 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:20.928515 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534669e5-f927-44f4-9de8-9647f692f475 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:20.928515 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534669e5-f927-44f4-9de8-9647f692f475 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:20.928515 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534669e5-f927-44f4-9de8-9647f692f475 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:20.928912 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534669e5-f927-44f4-9de8-9647f692f475 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:20.928912 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534669e5-f927-44f4-9de8-9647f692f475 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:20.965680 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e90d44-ff8e-427a-ab6a-94231f42e9be admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:20.968862 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-d9e90d44-ff8e-427a-ab6a-94231f42e9be admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:20.974471 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1036/2100] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:48:20 2022] POST /identity/v3/projects => generated 370 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:48:20.978067 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534669e5-f927-44f4-9de8-9647f692f475 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:20.986836 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e90d44-ff8e-427a-ab6a-94231f42e9be admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:21.004194 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=84,keystone:DELETE=13,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:21.088680 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0058ccde-d403-4241-a4c6-08179928809b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:21.089353 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0058ccde-d403-4241-a4c6-08179928809b admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:21.090280 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0058ccde-d403-4241-a4c6-08179928809b admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:21.090624 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0058ccde-d403-4241-a4c6-08179928809b admin admin] PATH_INFO: `/v3/users/4248ebe20e934d78800726fb766d7086` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:21.092256 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0058ccde-d403-4241-a4c6-08179928809b admin admin] RBAC: Authorizing `identity:delete_user(user_id=4248ebe20e934d78800726fb766d7086)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:21.095079 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0058ccde-d403-4241-a4c6-08179928809b admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:21.100679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534669e5-f927-44f4-9de8-9647f692f475 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:21.111884 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a12a1c-3b1d-424d-a06a-9a0e2b42392a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:21.112851 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a12a1c-3b1d-424d-a06a-9a0e2b42392a admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:21.113321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a12a1c-3b1d-424d-a06a-9a0e2b42392a admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:21.113782 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a12a1c-3b1d-424d-a06a-9a0e2b42392a admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:21.114730 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a12a1c-3b1d-424d-a06a-9a0e2b42392a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:21.118868 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a12a1c-3b1d-424d-a06a-9a0e2b42392a admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:21.149513 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1037/2101] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:48:20 2022] POST /identity/v3/users => generated 286 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:48:21.153846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a12a1c-3b1d-424d-a06a-9a0e2b42392a admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:21.167296 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a12a1c-3b1d-424d-a06a-9a0e2b42392a admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:21.174355 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-411acae9-06e9-4080-8aa4-895a66e05134 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:21.175021 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411acae9-06e9-4080-8aa4-895a66e05134 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:21.175221 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0058ccde-d403-4241-a4c6-08179928809b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4248ebe20e934d78800726fb766d7086'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:21.175344 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411acae9-06e9-4080-8aa4-895a66e05134 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:21.175568 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411acae9-06e9-4080-8aa4-895a66e05134 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:21.175931 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0058ccde-d403-4241-a4c6-08179928809b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4248ebe20e934d78800726fb766d7086'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:21.176310 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-411acae9-06e9-4080-8aa4-895a66e05134 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:21.176660 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0058ccde-d403-4241-a4c6-08179928809b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4248ebe20e934d78800726fb766d7086'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:21.177940 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-411acae9-06e9-4080-8aa4-895a66e05134 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:21.190323 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0058ccde-d403-4241-a4c6-08179928809b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4248ebe20e934d78800726fb766d7086'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:21.192158 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1038/2102] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:48:21 2022] GET /identity/v3/roles => generated 1427 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 15:48:21.197744 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-411acae9-06e9-4080-8aa4-895a66e05134 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:21.199701 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-411acae9-06e9-4080-8aa4-895a66e05134 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:21.212067 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0058ccde-d403-4241-a4c6-08179928809b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4248ebe20e934d78800726fb766d7086'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:21.213546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:21.214666 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-411acae9-06e9-4080-8aa4-895a66e05134 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:21.219813 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b7b4c8-4c04-4c6f-9951-31a585a040fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:21.220530 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b7b4c8-4c04-4c6f-9951-31a585a040fe admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:21.220886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b7b4c8-4c04-4c6f-9951-31a585a040fe admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:21.221278 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b7b4c8-4c04-4c6f-9951-31a585a040fe admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:21.222059 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b7b4c8-4c04-4c6f-9951-31a585a040fe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:21.224172 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b7b4c8-4c04-4c6f-9951-31a585a040fe admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:21.231577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1065/2103] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:48:20 2022] DELETE /identity/v3/users/4248ebe20e934d78800726fb766d7086 => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:21.236257 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05101d16-663f-44fa-97cc-2a6d75d0ec09 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:21.238618 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05101d16-663f-44fa-97cc-2a6d75d0ec09 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:21.239167 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05101d16-663f-44fa-97cc-2a6d75d0ec09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:21.245523 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1039/2104] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:48:21 2022] GET /identity/v3/roles => generated 1427 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 15:48:21.246542 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b7b4c8-4c04-4c6f-9951-31a585a040fe admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:21.258347 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b7b4c8-4c04-4c6f-9951-31a585a040fe admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:21.267244 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-05101d16-663f-44fa-97cc-2a6d75d0ec09 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:21.269013 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-05101d16-663f-44fa-97cc-2a6d75d0ec09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:21.331381 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e499fb1-f43f-43c2-b96a-455c3eb6d1e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:21.332409 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e499fb1-f43f-43c2-b96a-455c3eb6d1e2 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:21.332762 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e499fb1-f43f-43c2-b96a-455c3eb6d1e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:21.333115 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e499fb1-f43f-43c2-b96a-455c3eb6d1e2 admin admin] PATH_INFO: `/v3/projects/35c2600c3c96419494104b5ed625ad36/users/42547876bd24451e9186841395521cfa/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:21.352711 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e499fb1-f43f-43c2-b96a-455c3eb6d1e2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=35c2600c3c96419494104b5ed625ad36, user_id=42547876bd24451e9186841395521cfa, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:21.356578 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1066/2105] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:21 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:48:21.356674 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e499fb1-f43f-43c2-b96a-455c3eb6d1e2 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:21.372605 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05101d16-663f-44fa-97cc-2a6d75d0ec09 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:21.374824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-05101d16-663f-44fa-97cc-2a6d75d0ec09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:21.375704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1040/2106] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:48:21 2022] PUT /identity/v3/projects/35c2600c3c96419494104b5ed625ad36/users/42547876bd24451e9186841395521cfa/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:21.381266 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7575bce3-29cc-4c36-b9f8-d4bc1cca87e8 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:21.382195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7575bce3-29cc-4c36-b9f8-d4bc1cca87e8 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:21.383073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7575bce3-29cc-4c36-b9f8-d4bc1cca87e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:21.419620 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-7575bce3-29cc-4c36-b9f8-d4bc1cca87e8 None None] MFA Rules not processed for user `42547876bd24451e9186841395521cfa`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:21.421042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-7575bce3-29cc-4c36-b9f8-d4bc1cca87e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:21.448032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05101d16-663f-44fa-97cc-2a6d75d0ec09 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:21.495950 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:21.519244 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1041/2107] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:21 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:21.520190 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7575bce3-29cc-4c36-b9f8-d4bc1cca87e8 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:21.526049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-7575bce3-29cc-4c36-b9f8-d4bc1cca87e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:21.530233 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f234dfe6-6cee-46b9-8d3c-6194fbc464ef service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:21.531042 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f234dfe6-6cee-46b9-8d3c-6194fbc464ef service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:21.531389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f234dfe6-6cee-46b9-8d3c-6194fbc464ef service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:21.548295 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f234dfe6-6cee-46b9-8d3c-6194fbc464ef service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:21.638288 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:21.641536 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f234dfe6-6cee-46b9-8d3c-6194fbc464ef service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:21.643465 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f234dfe6-6cee-46b9-8d3c-6194fbc464ef service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:21.647544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7575bce3-29cc-4c36-b9f8-d4bc1cca87e8 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:21.653802 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8372287e-2f97-454d-8a94-23f7addf1e93 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c6f06186c1a4626997db5a40c31cbb9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:21.654426 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8372287e-2f97-454d-8a94-23f7addf1e93 service glance] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:21.654708 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8372287e-2f97-454d-8a94-23f7addf1e93 service glance] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:21.655015 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8372287e-2f97-454d-8a94-23f7addf1e93 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:21.658608 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1067/2108] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:48:21 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 287 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:48:21.667298 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8372287e-2f97-454d-8a94-23f7addf1e93 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:21.669143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8372287e-2f97-454d-8a94-23f7addf1e93 service glance] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:21.680004 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1042/2109] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:48:21 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:48:21.989203 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1a719f-7027-4f23-b9fd-af3c7f21187d None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:21.990349 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1a719f-7027-4f23-b9fd-af3c7f21187d None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:21.990825 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1a719f-7027-4f23-b9fd-af3c7f21187d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:22.013399 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-6c1a719f-7027-4f23-b9fd-af3c7f21187d None None] MFA Rules not processed for user `1e34111a3b2f4c1fb75252c4806d3da4`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:22.015261 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-6c1a719f-7027-4f23-b9fd-af3c7f21187d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:22.109958 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1068/2110] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:21 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:22.447872 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8372287e-2f97-454d-8a94-23f7addf1e93 service glance] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:22.449252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-8372287e-2f97-454d-8a94-23f7addf1e93 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:22.464233 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8372287e-2f97-454d-8a94-23f7addf1e93 service glance] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:22.468234 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd51efdc-6133-4ffb-88c3-9768a3d4c33d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:22.468868 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd51efdc-6133-4ffb-88c3-9768a3d4c33d admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:22.469145 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd51efdc-6133-4ffb-88c3-9768a3d4c33d admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:22.469413 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd51efdc-6133-4ffb-88c3-9768a3d4c33d admin admin] PATH_INFO: `/v3/projects/eb6a9b296ce74b569d52df41c8449820` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:22.470580 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd51efdc-6133-4ffb-88c3-9768a3d4c33d admin admin] RBAC: Authorizing `identity:delete_project(project_id=eb6a9b296ce74b569d52df41c8449820)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:22.472676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd51efdc-6133-4ffb-88c3-9768a3d4c33d admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:22.569584 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-bd51efdc-6133-4ffb-88c3-9768a3d4c33d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eb6a9b296ce74b569d52df41c8449820', 'request_id': 'req-bd51efdc-6133-4ffb-88c3-9768a3d4c33d'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:22.579134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-bd51efdc-6133-4ffb-88c3-9768a3d4c33d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eb6a9b296ce74b569d52df41c8449820', 'request_id': 'req-bd51efdc-6133-4ffb-88c3-9768a3d4c33d'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:22.587210 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-bd51efdc-6133-4ffb-88c3-9768a3d4c33d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eb6a9b296ce74b569d52df41c8449820', 'request_id': 'req-bd51efdc-6133-4ffb-88c3-9768a3d4c33d'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:22.627890 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-bd51efdc-6133-4ffb-88c3-9768a3d4c33d admin admin] The token cache is being invalidate because project eb6a9b296ce74b569d52df41c8449820 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:48:22.627890 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-bd51efdc-6133-4ffb-88c3-9768a3d4c33d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:22.627890 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-bd51efdc-6133-4ffb-88c3-9768a3d4c33d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:22.627890 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1043/2111] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:48:22 2022] DELETE /identity/v3/projects/eb6a9b296ce74b569d52df41c8449820 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:22.633794 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f795c5f9-fad5-4e8e-84b2-515fd12d56f6 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:22.634298 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f795c5f9-fad5-4e8e-84b2-515fd12d56f6 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:22.634769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f795c5f9-fad5-4e8e-84b2-515fd12d56f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:22.664687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-f795c5f9-fad5-4e8e-84b2-515fd12d56f6 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:22.666432 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-f795c5f9-fad5-4e8e-84b2-515fd12d56f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:22.748106 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1069/2112] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:22 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:22.756690 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f04493-e9a0-4782-a36f-36064ba822bf None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:22.756998 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f04493-e9a0-4782-a36f-36064ba822bf None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:22.757293 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f04493-e9a0-4782-a36f-36064ba822bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:22.782517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-c6f04493-e9a0-4782-a36f-36064ba822bf None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:22.783872 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-c6f04493-e9a0-4782-a36f-36064ba822bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:22.793737 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1044/2113] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:22 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:22.797844 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f795c5f9-fad5-4e8e-84b2-515fd12d56f6 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:22.809565 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f795c5f9-fad5-4e8e-84b2-515fd12d56f6 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:22.817200 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-132568ff-cbc6-4b3e-a49b-902982945e0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:22.817925 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132568ff-cbc6-4b3e-a49b-902982945e0e admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:22.818839 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132568ff-cbc6-4b3e-a49b-902982945e0e admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:22.819386 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132568ff-cbc6-4b3e-a49b-902982945e0e admin admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:22.820137 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-132568ff-cbc6-4b3e-a49b-902982945e0e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:48:22.821430 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-132568ff-cbc6-4b3e-a49b-902982945e0e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:22.825322 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-132568ff-cbc6-4b3e-a49b-902982945e0e admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:22.839714 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1070/2114] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:48:22 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:48:22.856646 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f04493-e9a0-4782-a36f-36064ba822bf None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:22.870915 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f04493-e9a0-4782-a36f-36064ba822bf None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:22.876220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021d1385-2d5c-4c94-a1bb-28f26c4702e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:22.876842 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021d1385-2d5c-4c94-a1bb-28f26c4702e9 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:22.877791 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021d1385-2d5c-4c94-a1bb-28f26c4702e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:22.878074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021d1385-2d5c-4c94-a1bb-28f26c4702e9 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:22.878725 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021d1385-2d5c-4c94-a1bb-28f26c4702e9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:22.880689 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021d1385-2d5c-4c94-a1bb-28f26c4702e9 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:22.926908 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1045/2115] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:48:22 2022] POST /identity/v3/projects => generated 376 bytes in 71 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:48:22.931609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-132568ff-cbc6-4b3e-a49b-902982945e0e admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:22.942842 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-132568ff-cbc6-4b3e-a49b-902982945e0e admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:23.017001 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2dbb89c-8480-4243-ac29-64c0adf9fcb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:23.017687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2dbb89c-8480-4243-ac29-64c0adf9fcb7 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:23.017900 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2dbb89c-8480-4243-ac29-64c0adf9fcb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:23.018171 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2dbb89c-8480-4243-ac29-64c0adf9fcb7 admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:23.018810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2dbb89c-8480-4243-ac29-64c0adf9fcb7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:23.020785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2dbb89c-8480-4243-ac29-64c0adf9fcb7 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:23.047946 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1071/2116] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:48:22 2022] POST /identity/v3/users => generated 289 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:48:23.049583 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021d1385-2d5c-4c94-a1bb-28f26c4702e9 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:23.061408 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021d1385-2d5c-4c94-a1bb-28f26c4702e9 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:23.066204 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0d44cb-82a1-4fb3-8937-16b3e670078b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:23.066806 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0d44cb-82a1-4fb3-8937-16b3e670078b admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:23.067086 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0d44cb-82a1-4fb3-8937-16b3e670078b admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:23.067359 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0d44cb-82a1-4fb3-8937-16b3e670078b admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:23.068044 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a0d44cb-82a1-4fb3-8937-16b3e670078b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:23.069907 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a0d44cb-82a1-4fb3-8937-16b3e670078b admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:23.081342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1046/2117] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:48:23 2022] GET /identity/v3/roles => generated 1427 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 15:48:23.086913 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2dbb89c-8480-4243-ac29-64c0adf9fcb7 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:23.101857 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2dbb89c-8480-4243-ac29-64c0adf9fcb7 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:23.107433 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b2316b-4e64-4468-9a54-d82f50541a8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:23.108583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b2316b-4e64-4468-9a54-d82f50541a8d admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:23.109041 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b2316b-4e64-4468-9a54-d82f50541a8d admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:23.109356 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b2316b-4e64-4468-9a54-d82f50541a8d admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:23.110189 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b2316b-4e64-4468-9a54-d82f50541a8d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:23.112646 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b2316b-4e64-4468-9a54-d82f50541a8d admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:23.130879 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1072/2118] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:48:23 2022] GET /identity/v3/roles => generated 1427 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:48:23.135137 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0d44cb-82a1-4fb3-8937-16b3e670078b admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:23.146823 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0d44cb-82a1-4fb3-8937-16b3e670078b admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:23.152943 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13cac722-6789-4d68-8a5b-6fd9c4eb2875 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:23.153785 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13cac722-6789-4d68-8a5b-6fd9c4eb2875 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:23.154068 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13cac722-6789-4d68-8a5b-6fd9c4eb2875 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:23.155559 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13cac722-6789-4d68-8a5b-6fd9c4eb2875 admin admin] PATH_INFO: `/v3/projects/e159b04c01254c0ca76aa169cffbff71/users/8b825cf40af64efaa17434f3337c3203/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:23.172650 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13cac722-6789-4d68-8a5b-6fd9c4eb2875 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e159b04c01254c0ca76aa169cffbff71, user_id=8b825cf40af64efaa17434f3337c3203, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:23.175412 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13cac722-6789-4d68-8a5b-6fd9c4eb2875 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:23.179039 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:23.209567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1047/2119] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:48:23 2022] PUT /identity/v3/projects/e159b04c01254c0ca76aa169cffbff71/users/8b825cf40af64efaa17434f3337c3203/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:23.211491 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9b243a-5f84-4e0a-9711-e4b6bcba5854 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:23.211814 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9b243a-5f84-4e0a-9711-e4b6bcba5854 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:23.212098 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9b243a-5f84-4e0a-9711-e4b6bcba5854 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:23.229793 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:23.255981 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-3c9b243a-5f84-4e0a-9711-e4b6bcba5854 None None] MFA Rules not processed for user `8b825cf40af64efaa17434f3337c3203`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:23.258422 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-3c9b243a-5f84-4e0a-9711-e4b6bcba5854 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:23.365304 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1073/2120] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:23 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:23.380575 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13cac722-6789-4d68-8a5b-6fd9c4eb2875 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:23.385739 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-13cac722-6789-4d68-8a5b-6fd9c4eb2875 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:23.467203 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13cac722-6789-4d68-8a5b-6fd9c4eb2875 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:23.473015 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0a104c-1836-4151-9d27-cda7a6d7780b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:23.473852 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0a104c-1836-4151-9d27-cda7a6d7780b service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:23.474281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0a104c-1836-4151-9d27-cda7a6d7780b service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:23.474699 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0a104c-1836-4151-9d27-cda7a6d7780b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:23.486720 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0a104c-1836-4151-9d27-cda7a6d7780b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:23.488352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0a104c-1836-4151-9d27-cda7a6d7780b service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:23.500312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1048/2121] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:48:23 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 15:48:26.149513 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c9b243a-5f84-4e0a-9711-e4b6bcba5854 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:26.152237 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-3c9b243a-5f84-4e0a-9711-e4b6bcba5854 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:26.233058 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c9b243a-5f84-4e0a-9711-e4b6bcba5854 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:26.237626 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a15457-c686-47f7-84bf-74d86493de90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:26.238263 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a15457-c686-47f7-84bf-74d86493de90 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:26.238533 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a15457-c686-47f7-84bf-74d86493de90 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:26.238803 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a15457-c686-47f7-84bf-74d86493de90 admin admin] PATH_INFO: `/v3/users/9db03e18a7b24157a8ac724a615373f4` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:26.239960 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a15457-c686-47f7-84bf-74d86493de90 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9db03e18a7b24157a8ac724a615373f4)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:26.242472 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a15457-c686-47f7-84bf-74d86493de90 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:26.293804 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-24a15457-c686-47f7-84bf-74d86493de90 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9db03e18a7b24157a8ac724a615373f4'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:26.294523 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-24a15457-c686-47f7-84bf-74d86493de90 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9db03e18a7b24157a8ac724a615373f4'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:26.295308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-24a15457-c686-47f7-84bf-74d86493de90 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9db03e18a7b24157a8ac724a615373f4'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:26.304251 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-24a15457-c686-47f7-84bf-74d86493de90 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9db03e18a7b24157a8ac724a615373f4'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:26.317524 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-24a15457-c686-47f7-84bf-74d86493de90 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9db03e18a7b24157a8ac724a615373f4'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:26.337917 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1074/2122] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:48:26 2022] DELETE /identity/v3/users/9db03e18a7b24157a8ac724a615373f4 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:26.541392 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0a104c-1836-4151-9d27-cda7a6d7780b service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:26.560977 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0a104c-1836-4151-9d27-cda7a6d7780b service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:26.642408 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183c26e8-6322-4b5f-bd72-4a8310580ca3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:26.643064 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183c26e8-6322-4b5f-bd72-4a8310580ca3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:26.643346 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183c26e8-6322-4b5f-bd72-4a8310580ca3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:26.656311 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183c26e8-6322-4b5f-bd72-4a8310580ca3 admin admin] PATH_INFO: `/v3/projects/b454a2613ce5453b81b0a78019eed614` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:26.656311 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183c26e8-6322-4b5f-bd72-4a8310580ca3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b454a2613ce5453b81b0a78019eed614)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:26.656311 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183c26e8-6322-4b5f-bd72-4a8310580ca3 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:26.734131 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-183c26e8-6322-4b5f-bd72-4a8310580ca3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b454a2613ce5453b81b0a78019eed614', 'request_id': 'req-183c26e8-6322-4b5f-bd72-4a8310580ca3'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:26.740715 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-183c26e8-6322-4b5f-bd72-4a8310580ca3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b454a2613ce5453b81b0a78019eed614', 'request_id': 'req-183c26e8-6322-4b5f-bd72-4a8310580ca3'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:26.747259 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-183c26e8-6322-4b5f-bd72-4a8310580ca3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b454a2613ce5453b81b0a78019eed614', 'request_id': 'req-183c26e8-6322-4b5f-bd72-4a8310580ca3'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:26.776352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-183c26e8-6322-4b5f-bd72-4a8310580ca3 admin admin] The token cache is being invalidate because project b454a2613ce5453b81b0a78019eed614 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:48:26.776650 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-183c26e8-6322-4b5f-bd72-4a8310580ca3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:26.777364 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-183c26e8-6322-4b5f-bd72-4a8310580ca3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:26.778989 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1049/2123] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:48:26 2022] DELETE /identity/v3/projects/b454a2613ce5453b81b0a78019eed614 => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:26.784000 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb8c7dc-573f-42bd-8675-4682326d463c None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:26.784391 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb8c7dc-573f-42bd-8675-4682326d463c None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:26.784783 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb8c7dc-573f-42bd-8675-4682326d463c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:26.813874 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-8bb8c7dc-573f-42bd-8675-4682326d463c None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:26.815182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-8bb8c7dc-573f-42bd-8675-4682326d463c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:26.837942 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:26.968677 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1075/2124] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:26 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:26.975938 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7849003b-138c-4535-a735-42b6de98f3a0 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:26.976928 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7849003b-138c-4535-a735-42b6de98f3a0 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:26.977173 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7849003b-138c-4535-a735-42b6de98f3a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:27.017291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-7849003b-138c-4535-a735-42b6de98f3a0 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:27.023941 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-7849003b-138c-4535-a735-42b6de98f3a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:27.032708 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1050/2125] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:26 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:27.037137 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb8c7dc-573f-42bd-8675-4682326d463c None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:27.049711 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb8c7dc-573f-42bd-8675-4682326d463c None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:27.055345 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4fefa6-6717-43e0-89eb-e89c8f851f2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:27.056236 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4fefa6-6717-43e0-89eb-e89c8f851f2b admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:27.056510 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4fefa6-6717-43e0-89eb-e89c8f851f2b admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:27.056932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4fefa6-6717-43e0-89eb-e89c8f851f2b admin admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:27.057734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4fefa6-6717-43e0-89eb-e89c8f851f2b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:48:27.058533 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4fefa6-6717-43e0-89eb-e89c8f851f2b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:27.060223 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4fefa6-6717-43e0-89eb-e89c8f851f2b admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:27.090709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1076/2126] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:48:27 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:48:27.094731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7849003b-138c-4535-a735-42b6de98f3a0 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:27.122804 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7849003b-138c-4535-a735-42b6de98f3a0 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:27.129080 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c921d2-0933-4008-9045-c718918d34fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:27.129715 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c921d2-0933-4008-9045-c718918d34fd admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:27.129998 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c921d2-0933-4008-9045-c718918d34fd admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:27.130269 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c921d2-0933-4008-9045-c718918d34fd admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:27.130901 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c921d2-0933-4008-9045-c718918d34fd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:27.133008 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c921d2-0933-4008-9045-c718918d34fd admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:27.167325 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:27.190681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1051/2127] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:48:27 2022] POST /identity/v3/projects => generated 384 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:48:27.192897 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4fefa6-6717-43e0-89eb-e89c8f851f2b admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:27.206726 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4fefa6-6717-43e0-89eb-e89c8f851f2b admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:27.281787 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71c42f9-432f-48e2-b830-83be2e577bb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:27.282413 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71c42f9-432f-48e2-b830-83be2e577bb3 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:27.282692 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71c42f9-432f-48e2-b830-83be2e577bb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:27.282966 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71c42f9-432f-48e2-b830-83be2e577bb3 admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:27.283597 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a71c42f9-432f-48e2-b830-83be2e577bb3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:27.290869 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a71c42f9-432f-48e2-b830-83be2e577bb3 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:27.331805 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1077/2128] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:48:27 2022] POST /identity/v3/users => generated 293 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 15:48:27.333470 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c921d2-0933-4008-9045-c718918d34fd admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:27.344916 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c921d2-0933-4008-9045-c718918d34fd admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:27.349679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2439872-8ffe-419a-9ceb-3bc9058c909f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:27.350289 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2439872-8ffe-419a-9ceb-3bc9058c909f admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:27.350565 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2439872-8ffe-419a-9ceb-3bc9058c909f admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:27.350835 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2439872-8ffe-419a-9ceb-3bc9058c909f admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:27.351710 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2439872-8ffe-419a-9ceb-3bc9058c909f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:27.353343 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2439872-8ffe-419a-9ceb-3bc9058c909f admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:27.365445 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1052/2129] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:48:27 2022] GET /identity/v3/roles => generated 1427 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:48:27.369973 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71c42f9-432f-48e2-b830-83be2e577bb3 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:27.380019 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71c42f9-432f-48e2-b830-83be2e577bb3 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:27.386834 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c178b9-60f9-4a19-a06a-41b75ac49f90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:27.387248 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c178b9-60f9-4a19-a06a-41b75ac49f90 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:27.387620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c178b9-60f9-4a19-a06a-41b75ac49f90 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:27.388010 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c178b9-60f9-4a19-a06a-41b75ac49f90 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:27.388937 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c178b9-60f9-4a19-a06a-41b75ac49f90 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:27.390466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c178b9-60f9-4a19-a06a-41b75ac49f90 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:27.411855 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1078/2130] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:48:27 2022] GET /identity/v3/roles => generated 1427 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:48:27.413073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2439872-8ffe-419a-9ceb-3bc9058c909f admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:27.427229 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2439872-8ffe-419a-9ceb-3bc9058c909f admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:27.432950 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be93f702-c79b-4c06-8b60-531f13b31f34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:27.434133 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be93f702-c79b-4c06-8b60-531f13b31f34 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:27.434133 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be93f702-c79b-4c06-8b60-531f13b31f34 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:27.434133 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be93f702-c79b-4c06-8b60-531f13b31f34 admin admin] PATH_INFO: `/v3/projects/f7f37ef7959d4674acca179a4797e272/users/ee49fabdcd304c72afefa48a25644bae/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:27.459270 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be93f702-c79b-4c06-8b60-531f13b31f34 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f7f37ef7959d4674acca179a4797e272, user_id=ee49fabdcd304c72afefa48a25644bae, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:27.463488 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be93f702-c79b-4c06-8b60-531f13b31f34 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:27.485987 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1053/2131] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:48:27 2022] PUT /identity/v3/projects/f7f37ef7959d4674acca179a4797e272/users/ee49fabdcd304c72afefa48a25644bae/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:27.491252 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99d808e-dc64-465f-86ee-6424672e4ca8 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:27.491619 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99d808e-dc64-465f-86ee-6424672e4ca8 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:27.492804 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99d808e-dc64-465f-86ee-6424672e4ca8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:27.513559 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-e99d808e-dc64-465f-86ee-6424672e4ca8 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:27.514930 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-e99d808e-dc64-465f-86ee-6424672e4ca8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:27.585778 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1079/2132] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:27 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:27.595765 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be93f702-c79b-4c06-8b60-531f13b31f34 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:27.597957 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-be93f702-c79b-4c06-8b60-531f13b31f34 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:27.665079 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be93f702-c79b-4c06-8b60-531f13b31f34 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:27.668842 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431c1524-25af-4cba-84e6-8e99d2dc278b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:27.669502 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431c1524-25af-4cba-84e6-8e99d2dc278b service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:27.670117 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431c1524-25af-4cba-84e6-8e99d2dc278b service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:27.670395 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431c1524-25af-4cba-84e6-8e99d2dc278b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:27.686871 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-431c1524-25af-4cba-84e6-8e99d2dc278b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:27.688872 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-431c1524-25af-4cba-84e6-8e99d2dc278b service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:27.701838 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1054/2133] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:48:27 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:48:33.214980 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c963df30-b619-4e48-9fd6-408d38668d69 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:33.215567 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c963df30-b619-4e48-9fd6-408d38668d69 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:33.215567 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c963df30-b619-4e48-9fd6-408d38668d69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:33.264910 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-c963df30-b619-4e48-9fd6-408d38668d69 None None] MFA Rules not processed for user `ee49fabdcd304c72afefa48a25644bae`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:33.266779 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-c963df30-b619-4e48-9fd6-408d38668d69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:33.283866 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:33.381272 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1080/2134] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:33 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:48:33.396787 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431c1524-25af-4cba-84e6-8e99d2dc278b service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:33.398291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-431c1524-25af-4cba-84e6-8e99d2dc278b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:33.479213 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431c1524-25af-4cba-84e6-8e99d2dc278b service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:33.485069 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a915b911-3cf5-436a-92e9-b313a4bc8e87 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:33.485679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a915b911-3cf5-436a-92e9-b313a4bc8e87 service nova] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:33.485959 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a915b911-3cf5-436a-92e9-b313a4bc8e87 service nova] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:33.486233 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a915b911-3cf5-436a-92e9-b313a4bc8e87 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:33.495340 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a915b911-3cf5-436a-92e9-b313a4bc8e87 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:33.496960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a915b911-3cf5-436a-92e9-b313a4bc8e87 service nova] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:33.505707 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1055/2135] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:48:33 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:48:34.983283 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c963df30-b619-4e48-9fd6-408d38668d69 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:34.985015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-c963df30-b619-4e48-9fd6-408d38668d69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:35.002214 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c963df30-b619-4e48-9fd6-408d38668d69 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:35.007862 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da1540b-a25a-4aab-9796-33607c56a013 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:35.008715 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da1540b-a25a-4aab-9796-33607c56a013 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:35.009164 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da1540b-a25a-4aab-9796-33607c56a013 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:35.009709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da1540b-a25a-4aab-9796-33607c56a013 admin admin] PATH_INFO: `/v3/users/8b825cf40af64efaa17434f3337c3203` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:35.011056 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da1540b-a25a-4aab-9796-33607c56a013 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8b825cf40af64efaa17434f3337c3203)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:35.014327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da1540b-a25a-4aab-9796-33607c56a013 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:35.016610 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8bdbea-7d0e-49ad-af83-1da8e9dd2d96 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:35.017509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8bdbea-7d0e-49ad-af83-1da8e9dd2d96 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:35.017779 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8bdbea-7d0e-49ad-af83-1da8e9dd2d96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:35.041343 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-4f8bdbea-7d0e-49ad-af83-1da8e9dd2d96 None None] MFA Rules not processed for user `ee49fabdcd304c72afefa48a25644bae`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:35.044147 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-4f8bdbea-7d0e-49ad-af83-1da8e9dd2d96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:35.052686 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1056/2136] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:35 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:35.101102 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4da1540b-a25a-4aab-9796-33607c56a013 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8b825cf40af64efaa17434f3337c3203'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:35.103839 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4da1540b-a25a-4aab-9796-33607c56a013 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8b825cf40af64efaa17434f3337c3203'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:35.104585 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4da1540b-a25a-4aab-9796-33607c56a013 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8b825cf40af64efaa17434f3337c3203'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:35.115020 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4da1540b-a25a-4aab-9796-33607c56a013 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8b825cf40af64efaa17434f3337c3203'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:35.143509 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4da1540b-a25a-4aab-9796-33607c56a013 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8b825cf40af64efaa17434f3337c3203'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:35.163474 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1081/2137] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:48:34 2022] DELETE /identity/v3/users/8b825cf40af64efaa17434f3337c3203 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:35.169783 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261cc9ff-47dd-4ffe-b73f-4fe89e8e93e8 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:35.170282 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261cc9ff-47dd-4ffe-b73f-4fe89e8e93e8 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:35.171552 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261cc9ff-47dd-4ffe-b73f-4fe89e8e93e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:35.206706 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-261cc9ff-47dd-4ffe-b73f-4fe89e8e93e8 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:35.208290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-261cc9ff-47dd-4ffe-b73f-4fe89e8e93e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:35.312968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:35.350772 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1057/2138] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:35 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:35.376066 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da1540b-a25a-4aab-9796-33607c56a013 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:35.378879 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-4da1540b-a25a-4aab-9796-33607c56a013 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:35.496196 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da1540b-a25a-4aab-9796-33607c56a013 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:35.500473 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a343f04-563c-4a05-8cb8-cc3cb347d169 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:35.501127 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a343f04-563c-4a05-8cb8-cc3cb347d169 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:35.501755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a343f04-563c-4a05-8cb8-cc3cb347d169 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:35.502454 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a343f04-563c-4a05-8cb8-cc3cb347d169 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:35.514757 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a343f04-563c-4a05-8cb8-cc3cb347d169 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:35.516277 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a343f04-563c-4a05-8cb8-cc3cb347d169 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:35.530140 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1082/2139] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:48:35 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:48:35.986912 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261cc9ff-47dd-4ffe-b73f-4fe89e8e93e8 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:35.988713 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-261cc9ff-47dd-4ffe-b73f-4fe89e8e93e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:36.004899 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261cc9ff-47dd-4ffe-b73f-4fe89e8e93e8 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:36.012348 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e89b201-6efd-4bc1-9068-7b2b97b3d067 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:36.013380 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e89b201-6efd-4bc1-9068-7b2b97b3d067 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:36.013959 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e89b201-6efd-4bc1-9068-7b2b97b3d067 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:36.014439 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e89b201-6efd-4bc1-9068-7b2b97b3d067 admin admin] PATH_INFO: `/v3/projects/e159b04c01254c0ca76aa169cffbff71` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:36.015924 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e89b201-6efd-4bc1-9068-7b2b97b3d067 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e159b04c01254c0ca76aa169cffbff71)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:36.018873 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e89b201-6efd-4bc1-9068-7b2b97b3d067 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:36.121958 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-4e89b201-6efd-4bc1-9068-7b2b97b3d067 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e159b04c01254c0ca76aa169cffbff71', 'request_id': 'req-4e89b201-6efd-4bc1-9068-7b2b97b3d067'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:36.136338 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-4e89b201-6efd-4bc1-9068-7b2b97b3d067 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e159b04c01254c0ca76aa169cffbff71', 'request_id': 'req-4e89b201-6efd-4bc1-9068-7b2b97b3d067'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:36.142514 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-4e89b201-6efd-4bc1-9068-7b2b97b3d067 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e159b04c01254c0ca76aa169cffbff71', 'request_id': 'req-4e89b201-6efd-4bc1-9068-7b2b97b3d067'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:36.171025 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-4e89b201-6efd-4bc1-9068-7b2b97b3d067 admin admin] The token cache is being invalidate because project e159b04c01254c0ca76aa169cffbff71 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:48:36.171719 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-4e89b201-6efd-4bc1-9068-7b2b97b3d067 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:36.172609 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-4e89b201-6efd-4bc1-9068-7b2b97b3d067 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:36.174368 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1058/2140] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:48:35 2022] DELETE /identity/v3/projects/e159b04c01254c0ca76aa169cffbff71 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:36.179863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e5cdf7-2a9a-486d-ba3b-3847bc83ad94 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:36.180974 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e5cdf7-2a9a-486d-ba3b-3847bc83ad94 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:36.181256 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e5cdf7-2a9a-486d-ba3b-3847bc83ad94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:36.202341 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-25e5cdf7-2a9a-486d-ba3b-3847bc83ad94 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:36.203558 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-25e5cdf7-2a9a-486d-ba3b-3847bc83ad94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:36.228594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:36.277919 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1083/2141] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:36 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:36.286911 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42cdfdf5-1e5c-4dd3-a266-afc5606578d6 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:36.287198 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42cdfdf5-1e5c-4dd3-a266-afc5606578d6 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:36.287572 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42cdfdf5-1e5c-4dd3-a266-afc5606578d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:36.316283 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-42cdfdf5-1e5c-4dd3-a266-afc5606578d6 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:36.317922 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-42cdfdf5-1e5c-4dd3-a266-afc5606578d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:36.330926 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1059/2142] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:36 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:36.332211 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e5cdf7-2a9a-486d-ba3b-3847bc83ad94 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:36.348062 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e5cdf7-2a9a-486d-ba3b-3847bc83ad94 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:36.362885 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e65c8c-b92c-41ea-a3d7-033d4aa9b1e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:36.363621 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e65c8c-b92c-41ea-a3d7-033d4aa9b1e6 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:36.363993 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e65c8c-b92c-41ea-a3d7-033d4aa9b1e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:36.364335 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e65c8c-b92c-41ea-a3d7-033d4aa9b1e6 admin admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:36.365075 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e65c8c-b92c-41ea-a3d7-033d4aa9b1e6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:48:36.365454 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e65c8c-b92c-41ea-a3d7-033d4aa9b1e6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:36.367937 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e65c8c-b92c-41ea-a3d7-033d4aa9b1e6 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:36.386236 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1084/2143] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:48:36 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:48:36.394545 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42cdfdf5-1e5c-4dd3-a266-afc5606578d6 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:36.413224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42cdfdf5-1e5c-4dd3-a266-afc5606578d6 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:36.419684 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb83b96-5497-4bcb-8e8c-02068d0e5679 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:36.420373 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb83b96-5497-4bcb-8e8c-02068d0e5679 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:36.420714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb83b96-5497-4bcb-8e8c-02068d0e5679 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:36.420952 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb83b96-5497-4bcb-8e8c-02068d0e5679 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:36.421775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb83b96-5497-4bcb-8e8c-02068d0e5679 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:36.423531 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb83b96-5497-4bcb-8e8c-02068d0e5679 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:36.457107 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1060/2144] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:48:36 2022] POST /identity/v3/projects => generated 382 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:48:36.460950 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e65c8c-b92c-41ea-a3d7-033d4aa9b1e6 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:36.475139 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e65c8c-b92c-41ea-a3d7-033d4aa9b1e6 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:36.553482 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afcb830e-74b3-4119-b597-7b46426567fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:36.554121 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcb830e-74b3-4119-b597-7b46426567fd admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:36.555025 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcb830e-74b3-4119-b597-7b46426567fd admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:36.556012 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcb830e-74b3-4119-b597-7b46426567fd admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:36.556717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afcb830e-74b3-4119-b597-7b46426567fd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:36.558965 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afcb830e-74b3-4119-b597-7b46426567fd admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:36.586063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1085/2145] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:48:36 2022] POST /identity/v3/users => generated 292 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 15:48:36.589837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb83b96-5497-4bcb-8e8c-02068d0e5679 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:36.602048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb83b96-5497-4bcb-8e8c-02068d0e5679 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:36.607934 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0949ee2b-9c61-4a1a-bc78-99f3c60ec8e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:36.608547 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0949ee2b-9c61-4a1a-bc78-99f3c60ec8e7 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:36.608829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0949ee2b-9c61-4a1a-bc78-99f3c60ec8e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:36.609260 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0949ee2b-9c61-4a1a-bc78-99f3c60ec8e7 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:36.609919 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0949ee2b-9c61-4a1a-bc78-99f3c60ec8e7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:36.611727 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0949ee2b-9c61-4a1a-bc78-99f3c60ec8e7 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:36.623564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1061/2146] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:48:36 2022] GET /identity/v3/roles => generated 1427 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:48:36.627649 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afcb830e-74b3-4119-b597-7b46426567fd admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:36.639489 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afcb830e-74b3-4119-b597-7b46426567fd admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:36.645851 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90c95ad-5d12-4bed-84a4-317a1cf2734c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:36.646447 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90c95ad-5d12-4bed-84a4-317a1cf2734c admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:36.646905 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90c95ad-5d12-4bed-84a4-317a1cf2734c admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:36.647187 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90c95ad-5d12-4bed-84a4-317a1cf2734c admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:36.647878 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f90c95ad-5d12-4bed-84a4-317a1cf2734c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:36.650034 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f90c95ad-5d12-4bed-84a4-317a1cf2734c admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:36.665154 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1086/2147] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:48:36 2022] GET /identity/v3/roles => generated 1427 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 15:48:36.669294 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0949ee2b-9c61-4a1a-bc78-99f3c60ec8e7 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:36.684261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0949ee2b-9c61-4a1a-bc78-99f3c60ec8e7 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:36.697647 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a50200a-bb3b-4105-acaf-be56151cfd1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:36.698562 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a50200a-bb3b-4105-acaf-be56151cfd1b admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:36.698916 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a50200a-bb3b-4105-acaf-be56151cfd1b admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:36.699288 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a50200a-bb3b-4105-acaf-be56151cfd1b admin admin] PATH_INFO: `/v3/projects/c4b9035514b74d60b49d9d7e404d3b9d/users/127f047b72c443eab061a22b1155e053/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:36.716694 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a50200a-bb3b-4105-acaf-be56151cfd1b admin admin] RBAC: Authorizing `identity:create_grant(project_id=c4b9035514b74d60b49d9d7e404d3b9d, user_id=127f047b72c443eab061a22b1155e053, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:36.720244 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a50200a-bb3b-4105-acaf-be56151cfd1b admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:36.739405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1062/2148] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:48:36 2022] PUT /identity/v3/projects/c4b9035514b74d60b49d9d7e404d3b9d/users/127f047b72c443eab061a22b1155e053/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 15:48:36.746995 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fc66a6-9c86-4716-8b77-53f846329ee3 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:36.748346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fc66a6-9c86-4716-8b77-53f846329ee3 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:36.749016 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fc66a6-9c86-4716-8b77-53f846329ee3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:36.770123 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-03fc66a6-9c86-4716-8b77-53f846329ee3 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:36.771411 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-03fc66a6-9c86-4716-8b77-53f846329ee3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:36.853324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1087/2149] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:36 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:36.865213 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a50200a-bb3b-4105-acaf-be56151cfd1b admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:36.868663 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-2a50200a-bb3b-4105-acaf-be56151cfd1b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:37.027325 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a50200a-bb3b-4105-acaf-be56151cfd1b admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:37.033420 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4986eb1-bbf6-4738-9efa-13fe85317768 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:37.034045 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4986eb1-bbf6-4738-9efa-13fe85317768 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:37.034583 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4986eb1-bbf6-4738-9efa-13fe85317768 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:37.034864 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4986eb1-bbf6-4738-9efa-13fe85317768 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:37.048703 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4986eb1-bbf6-4738-9efa-13fe85317768 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:37.050190 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4986eb1-bbf6-4738-9efa-13fe85317768 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:37.062445 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1063/2150] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:48:36 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:48:38.052750 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d44d55f-6a92-4dad-8613-6ab4b0efffd7 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:38.053270 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d44d55f-6a92-4dad-8613-6ab4b0efffd7 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:38.053387 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d44d55f-6a92-4dad-8613-6ab4b0efffd7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:38.088251 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-2d44d55f-6a92-4dad-8613-6ab4b0efffd7 None None] MFA Rules not processed for user `127f047b72c443eab061a22b1155e053`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:38.089560 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-2d44d55f-6a92-4dad-8613-6ab4b0efffd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:38.120740 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:38.215418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1088/2151] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:38 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:38.252740 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4986eb1-bbf6-4738-9efa-13fe85317768 service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:38.254223 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-f4986eb1-bbf6-4738-9efa-13fe85317768 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:38.270766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:38.329280 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4986eb1-bbf6-4738-9efa-13fe85317768 service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:38.332984 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a461c3c-69ef-48fe-b027-00e5587965c0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:38.333837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a461c3c-69ef-48fe-b027-00e5587965c0 service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:38.334321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a461c3c-69ef-48fe-b027-00e5587965c0 service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:38.334776 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a461c3c-69ef-48fe-b027-00e5587965c0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:38.346983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a461c3c-69ef-48fe-b027-00e5587965c0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:38.348796 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a461c3c-69ef-48fe-b027-00e5587965c0 service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:38.380370 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1064/2152] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:48:38 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:48:39.868926 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5618b48c-92c4-43bc-a9d6-baed5070bf62 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:39.868926 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5618b48c-92c4-43bc-a9d6-baed5070bf62 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:39.868926 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5618b48c-92c4-43bc-a9d6-baed5070bf62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:39.889778 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-5618b48c-92c4-43bc-a9d6-baed5070bf62 None None] MFA Rules not processed for user `127f047b72c443eab061a22b1155e053`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:39.891167 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-5618b48c-92c4-43bc-a9d6-baed5070bf62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:39.901453 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1089/2153] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:39 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:44.027989 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a461c3c-69ef-48fe-b027-00e5587965c0 service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:44.029450 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-8a461c3c-69ef-48fe-b027-00e5587965c0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:44.043964 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a461c3c-69ef-48fe-b027-00e5587965c0 service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:44.048929 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7820f1f-fd27-4ae7-9f98-b9ee14a285c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:44.049548 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7820f1f-fd27-4ae7-9f98-b9ee14a285c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:44.049847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7820f1f-fd27-4ae7-9f98-b9ee14a285c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:44.050150 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7820f1f-fd27-4ae7-9f98-b9ee14a285c8 admin admin] PATH_INFO: `/v3/users/0b0053d3167e435db8b9a9be0beabfce` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:44.051450 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7820f1f-fd27-4ae7-9f98-b9ee14a285c8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0b0053d3167e435db8b9a9be0beabfce)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:44.053507 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7820f1f-fd27-4ae7-9f98-b9ee14a285c8 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:44.125406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-c7820f1f-fd27-4ae7-9f98-b9ee14a285c8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0b0053d3167e435db8b9a9be0beabfce'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:44.125997 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-c7820f1f-fd27-4ae7-9f98-b9ee14a285c8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0b0053d3167e435db8b9a9be0beabfce'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:44.133160 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-c7820f1f-fd27-4ae7-9f98-b9ee14a285c8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0b0053d3167e435db8b9a9be0beabfce'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:44.145135 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-c7820f1f-fd27-4ae7-9f98-b9ee14a285c8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0b0053d3167e435db8b9a9be0beabfce'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:44.145794 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-c7820f1f-fd27-4ae7-9f98-b9ee14a285c8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0b0053d3167e435db8b9a9be0beabfce'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:44.159417 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1065/2154] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:48:44 2022] DELETE /identity/v3/users/0b0053d3167e435db8b9a9be0beabfce => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:44.164589 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d943aa-9ba0-4210-91b7-f464e86f5551 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:44.165204 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d943aa-9ba0-4210-91b7-f464e86f5551 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:44.165204 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d943aa-9ba0-4210-91b7-f464e86f5551 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:44.183770 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-91d943aa-9ba0-4210-91b7-f464e86f5551 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:44.184960 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-91d943aa-9ba0-4210-91b7-f464e86f5551 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:44.244446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1090/2155] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:48:44 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:48:44.254182 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7820f1f-fd27-4ae7-9f98-b9ee14a285c8 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:44.255568 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-c7820f1f-fd27-4ae7-9f98-b9ee14a285c8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:44.314659 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7820f1f-fd27-4ae7-9f98-b9ee14a285c8 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:44.318343 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c292d74-db85-4c50-8bf6-aa1d5b9c068c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:44.318943 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c292d74-db85-4c50-8bf6-aa1d5b9c068c service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:44.319249 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c292d74-db85-4c50-8bf6-aa1d5b9c068c service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:44.319500 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c292d74-db85-4c50-8bf6-aa1d5b9c068c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:44.328861 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c292d74-db85-4c50-8bf6-aa1d5b9c068c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:44.330291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c292d74-db85-4c50-8bf6-aa1d5b9c068c service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:44.346894 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1066/2156] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:48:44 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:48:44.852279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d943aa-9ba0-4210-91b7-f464e86f5551 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:44.853839 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-91d943aa-9ba0-4210-91b7-f464e86f5551 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:44.869911 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d943aa-9ba0-4210-91b7-f464e86f5551 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:44.883173 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2a4bdd-67f5-4886-856b-0a47eb574762 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:44.884128 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2a4bdd-67f5-4886-856b-0a47eb574762 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:44.884583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2a4bdd-67f5-4886-856b-0a47eb574762 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:44.885012 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2a4bdd-67f5-4886-856b-0a47eb574762 admin admin] PATH_INFO: `/v3/projects/234a63a9132d4e908be0a928e25acd40` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:44.888918 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2a4bdd-67f5-4886-856b-0a47eb574762 admin admin] RBAC: Authorizing `identity:delete_project(project_id=234a63a9132d4e908be0a928e25acd40)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:44.890945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2a4bdd-67f5-4886-856b-0a47eb574762 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:44.944759 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-cf2a4bdd-67f5-4886-856b-0a47eb574762 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '234a63a9132d4e908be0a928e25acd40', 'request_id': 'req-cf2a4bdd-67f5-4886-856b-0a47eb574762'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:44.951461 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-cf2a4bdd-67f5-4886-856b-0a47eb574762 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '234a63a9132d4e908be0a928e25acd40', 'request_id': 'req-cf2a4bdd-67f5-4886-856b-0a47eb574762'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:44.957091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-cf2a4bdd-67f5-4886-856b-0a47eb574762 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '234a63a9132d4e908be0a928e25acd40', 'request_id': 'req-cf2a4bdd-67f5-4886-856b-0a47eb574762'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:44.971164 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-cf2a4bdd-67f5-4886-856b-0a47eb574762 admin admin] The token cache is being invalidate because project 234a63a9132d4e908be0a928e25acd40 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:48:44.971479 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-cf2a4bdd-67f5-4886-856b-0a47eb574762 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:44.972236 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-cf2a4bdd-67f5-4886-856b-0a47eb574762 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:48:44.975048 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1091/2157] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:48:44 2022] DELETE /identity/v3/projects/234a63a9132d4e908be0a928e25acd40 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:48:54.335443 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=77,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:54.968047 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=74,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:48:56.400007 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838ab7a5-2303-4e74-aa34-fcb6d3518b3b None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:56.400954 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838ab7a5-2303-4e74-aa34-fcb6d3518b3b None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:56.401430 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838ab7a5-2303-4e74-aa34-fcb6d3518b3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:56.422790 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-838ab7a5-2303-4e74-aa34-fcb6d3518b3b None None] MFA Rules not processed for user `a6c17620ead14cb49d962990e1b1e0d8`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:48:56.424319 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-838ab7a5-2303-4e74-aa34-fcb6d3518b3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:56.495219 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1067/2158] 10.222.0.102 () {58 vars in 1014 bytes} [Wed Aug 24 15:48:56 2022] POST /identity/v3/auth/tokens => generated 3205 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:48:56.557149 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2a4bdd-67f5-4886-856b-0a47eb574762 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:48:56.559186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-cf2a4bdd-67f5-4886-856b-0a47eb574762 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:48:56.644643 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2a4bdd-67f5-4886-856b-0a47eb574762 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:48:56.652104 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1dae2d5-3111-4f5b-a363-75343554010f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a070d4fd8c4466c857bd03965c11349', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:48:56.653077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1dae2d5-3111-4f5b-a363-75343554010f service swift] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:48:56.653450 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1dae2d5-3111-4f5b-a363-75343554010f service swift] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:48:56.654241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1dae2d5-3111-4f5b-a363-75343554010f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:48:56.668263 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1dae2d5-3111-4f5b-a363-75343554010f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:48:56.670074 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1dae2d5-3111-4f5b-a363-75343554010f service swift] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:48:56.682421 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1092/2159] 10.222.0.102 () {60 vars in 1389 bytes} [Wed Aug 24 15:48:56 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 24 15:49:06.487535 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=25 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:49:06.677298 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=32 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:49:10.602070 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838ab7a5-2303-4e74-aa34-fcb6d3518b3b None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:49:10.604348 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-838ab7a5-2303-4e74-aa34-fcb6d3518b3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:49:10.680007 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838ab7a5-2303-4e74-aa34-fcb6d3518b3b None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:49:10.684943 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0fac446-3c32-45f3-aae3-c4a6d6b5f274 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:49:10.685682 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0fac446-3c32-45f3-aae3-c4a6d6b5f274 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:10.686021 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0fac446-3c32-45f3-aae3-c4a6d6b5f274 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:10.686654 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0fac446-3c32-45f3-aae3-c4a6d6b5f274 admin admin] PATH_INFO: `/v3/users/42547876bd24451e9186841395521cfa` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:10.688009 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0fac446-3c32-45f3-aae3-c4a6d6b5f274 admin admin] RBAC: Authorizing `identity:delete_user(user_id=42547876bd24451e9186841395521cfa)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:49:10.690191 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0fac446-3c32-45f3-aae3-c4a6d6b5f274 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:49:10.755719 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-a0fac446-3c32-45f3-aae3-c4a6d6b5f274 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '42547876bd24451e9186841395521cfa'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:49:10.756525 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-a0fac446-3c32-45f3-aae3-c4a6d6b5f274 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '42547876bd24451e9186841395521cfa'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:49:10.772021 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-a0fac446-3c32-45f3-aae3-c4a6d6b5f274 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '42547876bd24451e9186841395521cfa'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:49:10.788247 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-a0fac446-3c32-45f3-aae3-c4a6d6b5f274 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '42547876bd24451e9186841395521cfa'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:49:10.789690 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-a0fac446-3c32-45f3-aae3-c4a6d6b5f274 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '42547876bd24451e9186841395521cfa'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:49:10.807144 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1068/2160] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:49:10 2022] DELETE /identity/v3/users/42547876bd24451e9186841395521cfa => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:49:10.811744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95b9939-136e-4ca5-8659-7936d4c7c139 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:10.812262 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95b9939-136e-4ca5-8659-7936d4c7c139 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:10.812350 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95b9939-136e-4ca5-8659-7936d4c7c139 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:10.838436 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-e95b9939-136e-4ca5-8659-7936d4c7c139 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:49:10.841758 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-e95b9939-136e-4ca5-8659-7936d4c7c139 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:49:10.927103 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1093/2161] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:49:10 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:49:10.938476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0fac446-3c32-45f3-aae3-c4a6d6b5f274 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:49:10.940525 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-a0fac446-3c32-45f3-aae3-c4a6d6b5f274 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:49:11.011056 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0fac446-3c32-45f3-aae3-c4a6d6b5f274 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:49:11.016329 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b20fd8-4524-43b9-b5bd-5e4093c1833c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:49:11.017063 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b20fd8-4524-43b9-b5bd-5e4093c1833c service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:11.017432 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b20fd8-4524-43b9-b5bd-5e4093c1833c service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:11.017790 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b20fd8-4524-43b9-b5bd-5e4093c1833c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:11.028602 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b20fd8-4524-43b9-b5bd-5e4093c1833c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:49:11.029928 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b20fd8-4524-43b9-b5bd-5e4093c1833c service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:49:11.041167 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1069/2162] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:49:10 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:49:11.432718 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95b9939-136e-4ca5-8659-7936d4c7c139 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:49:11.437042 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-e95b9939-136e-4ca5-8659-7936d4c7c139 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:49:11.462562 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95b9939-136e-4ca5-8659-7936d4c7c139 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:49:11.471296 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4a385c-5295-41a6-9d13-bfbb0c6dace4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:49:11.472352 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4a385c-5295-41a6-9d13-bfbb0c6dace4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:11.472641 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4a385c-5295-41a6-9d13-bfbb0c6dace4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:11.472999 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4a385c-5295-41a6-9d13-bfbb0c6dace4 admin admin] PATH_INFO: `/v3/projects/35c2600c3c96419494104b5ed625ad36` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:11.474222 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4a385c-5295-41a6-9d13-bfbb0c6dace4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=35c2600c3c96419494104b5ed625ad36)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:49:11.478060 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4a385c-5295-41a6-9d13-bfbb0c6dace4 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:49:11.538915 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4e4a385c-5295-41a6-9d13-bfbb0c6dace4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '35c2600c3c96419494104b5ed625ad36', 'request_id': 'req-4e4a385c-5295-41a6-9d13-bfbb0c6dace4'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:49:11.546215 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4e4a385c-5295-41a6-9d13-bfbb0c6dace4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '35c2600c3c96419494104b5ed625ad36', 'request_id': 'req-4e4a385c-5295-41a6-9d13-bfbb0c6dace4'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:49:11.553330 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4e4a385c-5295-41a6-9d13-bfbb0c6dace4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '35c2600c3c96419494104b5ed625ad36', 'request_id': 'req-4e4a385c-5295-41a6-9d13-bfbb0c6dace4'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:49:11.568006 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4e4a385c-5295-41a6-9d13-bfbb0c6dace4 admin admin] The token cache is being invalidate because project 35c2600c3c96419494104b5ed625ad36 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:49:11.568313 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4e4a385c-5295-41a6-9d13-bfbb0c6dace4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:49:11.568972 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4e4a385c-5295-41a6-9d13-bfbb0c6dace4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:49:11.571256 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1094/2163] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:49:11 2022] DELETE /identity/v3/projects/35c2600c3c96419494104b5ed625ad36 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:49:11.576487 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e5119e-3ef1-4b53-acd8-f5efb695886b None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:11.577080 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e5119e-3ef1-4b53-acd8-f5efb695886b None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:11.577496 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e5119e-3ef1-4b53-acd8-f5efb695886b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:11.602878 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-53e5119e-3ef1-4b53-acd8-f5efb695886b None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:49:11.604151 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-53e5119e-3ef1-4b53-acd8-f5efb695886b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:49:11.623110 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:49:11.670736 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1070/2164] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:49:11 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:49:11.679382 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e3866e-fda1-4cb3-9323-dc04f6a06a0d None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:11.679983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e3866e-fda1-4cb3-9323-dc04f6a06a0d None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:11.680283 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e3866e-fda1-4cb3-9323-dc04f6a06a0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:11.701883 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-62e3866e-fda1-4cb3-9323-dc04f6a06a0d None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:49:11.703219 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-62e3866e-fda1-4cb3-9323-dc04f6a06a0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:49:11.715608 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1095/2165] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:49:11 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:49:11.716983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e5119e-3ef1-4b53-acd8-f5efb695886b None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:49:11.727854 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e5119e-3ef1-4b53-acd8-f5efb695886b None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:49:11.732604 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27cb4779-da00-4699-abba-b2f892ea8785 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:49:11.733199 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cb4779-da00-4699-abba-b2f892ea8785 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:11.733622 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cb4779-da00-4699-abba-b2f892ea8785 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:11.733905 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cb4779-da00-4699-abba-b2f892ea8785 admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:11.734559 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27cb4779-da00-4699-abba-b2f892ea8785 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:49:11.734862 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27cb4779-da00-4699-abba-b2f892ea8785 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:49:11.736531 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27cb4779-da00-4699-abba-b2f892ea8785 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:49:11.750559 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1071/2166] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:49:11 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:49:11.755890 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e3866e-fda1-4cb3-9323-dc04f6a06a0d None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:49:11.770891 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e3866e-fda1-4cb3-9323-dc04f6a06a0d None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:49:11.777002 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e4a58c-9bda-4eaa-8109-a2abef884f6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:49:11.777629 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e4a58c-9bda-4eaa-8109-a2abef884f6a admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:11.777887 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e4a58c-9bda-4eaa-8109-a2abef884f6a admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:11.778161 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e4a58c-9bda-4eaa-8109-a2abef884f6a admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:11.778796 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e4a58c-9bda-4eaa-8109-a2abef884f6a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:49:11.780745 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e4a58c-9bda-4eaa-8109-a2abef884f6a admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:49:11.813178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1096/2167] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:49:11 2022] POST /identity/v3/projects => generated 372 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 15:49:11.817235 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27cb4779-da00-4699-abba-b2f892ea8785 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:49:11.828826 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27cb4779-da00-4699-abba-b2f892ea8785 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:49:11.904741 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080d45a8-3993-4ed8-bb5c-8c12239b72ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:49:11.905791 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080d45a8-3993-4ed8-bb5c-8c12239b72ed admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:11.906076 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080d45a8-3993-4ed8-bb5c-8c12239b72ed admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:11.906541 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080d45a8-3993-4ed8-bb5c-8c12239b72ed admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:11.907295 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080d45a8-3993-4ed8-bb5c-8c12239b72ed admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:49:11.909997 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080d45a8-3993-4ed8-bb5c-8c12239b72ed admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:49:11.938247 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1072/2168] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:49:11 2022] POST /identity/v3/users => generated 287 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:49:11.942089 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e4a58c-9bda-4eaa-8109-a2abef884f6a admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:49:11.954443 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e4a58c-9bda-4eaa-8109-a2abef884f6a admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:49:11.959184 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff68a626-813b-4c69-8939-ed6e0e509c43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:49:11.959819 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff68a626-813b-4c69-8939-ed6e0e509c43 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:11.960103 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff68a626-813b-4c69-8939-ed6e0e509c43 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:11.960376 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff68a626-813b-4c69-8939-ed6e0e509c43 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:11.961090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff68a626-813b-4c69-8939-ed6e0e509c43 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:49:11.962698 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff68a626-813b-4c69-8939-ed6e0e509c43 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:49:11.973586 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1097/2169] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:49:11 2022] GET /identity/v3/roles => generated 1427 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:49:11.978520 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080d45a8-3993-4ed8-bb5c-8c12239b72ed admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:49:11.993271 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080d45a8-3993-4ed8-bb5c-8c12239b72ed admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:49:11.999085 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b74937c-c7c8-4b05-8051-b11afd96251f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:49:11.999793 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b74937c-c7c8-4b05-8051-b11afd96251f admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:12.000078 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b74937c-c7c8-4b05-8051-b11afd96251f admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:12.000353 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b74937c-c7c8-4b05-8051-b11afd96251f admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:12.001009 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b74937c-c7c8-4b05-8051-b11afd96251f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:49:12.002653 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b74937c-c7c8-4b05-8051-b11afd96251f admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:49:12.016792 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1073/2170] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:49:11 2022] GET /identity/v3/roles => generated 1427 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:49:12.021339 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff68a626-813b-4c69-8939-ed6e0e509c43 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:49:12.033699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff68a626-813b-4c69-8939-ed6e0e509c43 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:49:12.038842 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0422c8f2-74c0-467a-b8b2-32ae4f00bac2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:49:12.040354 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0422c8f2-74c0-467a-b8b2-32ae4f00bac2 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:12.040651 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0422c8f2-74c0-467a-b8b2-32ae4f00bac2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:12.040938 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0422c8f2-74c0-467a-b8b2-32ae4f00bac2 admin admin] PATH_INFO: `/v3/projects/bd10d33165a345179b73fc117e1c084c/users/75117fe6092347c58cd3384c06eea72b/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:12.058414 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0422c8f2-74c0-467a-b8b2-32ae4f00bac2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bd10d33165a345179b73fc117e1c084c, user_id=75117fe6092347c58cd3384c06eea72b, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:49:12.061165 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0422c8f2-74c0-467a-b8b2-32ae4f00bac2 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:49:12.078883 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1098/2171] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:49:12 2022] PUT /identity/v3/projects/bd10d33165a345179b73fc117e1c084c/users/75117fe6092347c58cd3384c06eea72b/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 15:49:12.090227 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314c29ee-ceed-4d89-94f8-8ec48fd9ee15 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:12.090429 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314c29ee-ceed-4d89-94f8-8ec48fd9ee15 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:12.090698 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314c29ee-ceed-4d89-94f8-8ec48fd9ee15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:12.126075 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-314c29ee-ceed-4d89-94f8-8ec48fd9ee15 None None] MFA Rules not processed for user `75117fe6092347c58cd3384c06eea72b`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:49:12.127349 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-314c29ee-ceed-4d89-94f8-8ec48fd9ee15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:49:12.222497 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1074/2172] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:49:12 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:49:20.382162 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0422c8f2-74c0-467a-b8b2-32ae4f00bac2 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:49:20.385211 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-0422c8f2-74c0-467a-b8b2-32ae4f00bac2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:49:20.410504 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:49:20.472079 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0422c8f2-74c0-467a-b8b2-32ae4f00bac2 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:49:20.476683 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff49e2ea-04bf-43de-a298-3dce0d970e88 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c6f06186c1a4626997db5a40c31cbb9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:49:20.477328 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff49e2ea-04bf-43de-a298-3dce0d970e88 service glance] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:20.477613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff49e2ea-04bf-43de-a298-3dce0d970e88 service glance] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:20.478056 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff49e2ea-04bf-43de-a298-3dce0d970e88 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:20.489853 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff49e2ea-04bf-43de-a298-3dce0d970e88 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:49:20.491799 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff49e2ea-04bf-43de-a298-3dce0d970e88 service glance] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:49:20.501920 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1099/2173] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:49:20 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:49:22.210583 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=89,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:49:26.383991 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36973985-e042-4839-b08b-19c3abd86e28 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:26.385869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36973985-e042-4839-b08b-19c3abd86e28 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:26.385869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36973985-e042-4839-b08b-19c3abd86e28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:26.407222 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-36973985-e042-4839-b08b-19c3abd86e28 None None] MFA Rules not processed for user `a6c17620ead14cb49d962990e1b1e0d8`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:49:26.408327 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-36973985-e042-4839-b08b-19c3abd86e28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:49:26.491111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1075/2174] 10.222.0.102 () {58 vars in 972 bytes} [Wed Aug 24 15:49:26 2022] POST /identity/v3/auth/tokens => generated 3205 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:49:26.505113 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff49e2ea-04bf-43de-a298-3dce0d970e88 service glance] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:49:26.506747 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-ff49e2ea-04bf-43de-a298-3dce0d970e88 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:49:26.583694 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff49e2ea-04bf-43de-a298-3dce0d970e88 service glance] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:49:26.588815 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3c0d84-14f9-4dc1-9f0f-8680f1c66b98 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a070d4fd8c4466c857bd03965c11349', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:49:26.589428 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3c0d84-14f9-4dc1-9f0f-8680f1c66b98 service swift] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:26.589708 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3c0d84-14f9-4dc1-9f0f-8680f1c66b98 service swift] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:26.589981 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3c0d84-14f9-4dc1-9f0f-8680f1c66b98 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:26.601889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3c0d84-14f9-4dc1-9f0f-8680f1c66b98 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:49:26.604122 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3c0d84-14f9-4dc1-9f0f-8680f1c66b98 service swift] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:49:26.612583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1100/2175] 10.222.0.102 () {60 vars in 1389 bytes} [Wed Aug 24 15:49:26 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 24 15:49:27.401828 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3053b7-5774-4603-a62a-e1e02eb13046 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:27.402781 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3053b7-5774-4603-a62a-e1e02eb13046 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:27.403359 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3053b7-5774-4603-a62a-e1e02eb13046 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:27.427348 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-2f3053b7-5774-4603-a62a-e1e02eb13046 None None] MFA Rules not processed for user `a6c17620ead14cb49d962990e1b1e0d8`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:49:27.428733 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-2f3053b7-5774-4603-a62a-e1e02eb13046 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:49:27.437666 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1076/2176] 10.222.0.102 () {58 vars in 1014 bytes} [Wed Aug 24 15:49:27 2022] POST /identity/v3/auth/tokens => generated 3205 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:49:27.453009 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3c0d84-14f9-4dc1-9f0f-8680f1c66b98 service swift] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:49:27.467406 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3c0d84-14f9-4dc1-9f0f-8680f1c66b98 service swift] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:49:27.474833 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ece9a1-62f4-4620-be9d-3a1bf09c8974 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a070d4fd8c4466c857bd03965c11349', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:49:27.474833 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ece9a1-62f4-4620-be9d-3a1bf09c8974 service swift] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:27.474833 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ece9a1-62f4-4620-be9d-3a1bf09c8974 service swift] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:27.474833 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ece9a1-62f4-4620-be9d-3a1bf09c8974 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:27.485780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ece9a1-62f4-4620-be9d-3a1bf09c8974 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:49:27.487401 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ece9a1-62f4-4620-be9d-3a1bf09c8974 service swift] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:49:27.499490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1101/2177] 10.222.0.102 () {60 vars in 1389 bytes} [Wed Aug 24 15:49:27 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 24 15:49:28.946284 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3053b7-5774-4603-a62a-e1e02eb13046 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:49:28.948765 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-2f3053b7-5774-4603-a62a-e1e02eb13046 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:49:29.036113 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3053b7-5774-4603-a62a-e1e02eb13046 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:49:29.041647 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a2fc02-610c-4c7f-ac4b-7047d15093af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:49:29.042298 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a2fc02-610c-4c7f-ac4b-7047d15093af admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:29.042581 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a2fc02-610c-4c7f-ac4b-7047d15093af admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:29.042858 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a2fc02-610c-4c7f-ac4b-7047d15093af admin admin] PATH_INFO: `/v3/users/1e34111a3b2f4c1fb75252c4806d3da4` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:29.044191 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a2fc02-610c-4c7f-ac4b-7047d15093af admin admin] RBAC: Authorizing `identity:delete_user(user_id=1e34111a3b2f4c1fb75252c4806d3da4)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:49:29.046150 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a2fc02-610c-4c7f-ac4b-7047d15093af admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:49:29.109803 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-36a2fc02-610c-4c7f-ac4b-7047d15093af admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1e34111a3b2f4c1fb75252c4806d3da4'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:49:29.110592 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-36a2fc02-610c-4c7f-ac4b-7047d15093af admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1e34111a3b2f4c1fb75252c4806d3da4'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:49:29.120362 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-36a2fc02-610c-4c7f-ac4b-7047d15093af admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1e34111a3b2f4c1fb75252c4806d3da4'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:49:29.137102 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-36a2fc02-610c-4c7f-ac4b-7047d15093af admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1e34111a3b2f4c1fb75252c4806d3da4'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:49:29.138242 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-36a2fc02-610c-4c7f-ac4b-7047d15093af admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1e34111a3b2f4c1fb75252c4806d3da4'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:49:29.154174 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1077/2178] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:49:28 2022] DELETE /identity/v3/users/1e34111a3b2f4c1fb75252c4806d3da4 => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:49:29.350381 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ece9a1-62f4-4620-be9d-3a1bf09c8974 service swift] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:49:29.385134 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ece9a1-62f4-4620-be9d-3a1bf09c8974 service swift] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:49:29.492095 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d2b1f0-0266-482a-a845-6b1098e1bd53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:49:29.493064 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d2b1f0-0266-482a-a845-6b1098e1bd53 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:29.493414 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d2b1f0-0266-482a-a845-6b1098e1bd53 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:29.493808 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d2b1f0-0266-482a-a845-6b1098e1bd53 admin admin] PATH_INFO: `/v3/projects/26e324399ad64863a994246cc04a4751` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:29.495805 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d2b1f0-0266-482a-a845-6b1098e1bd53 admin admin] RBAC: Authorizing `identity:delete_project(project_id=26e324399ad64863a994246cc04a4751)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:49:29.498031 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d2b1f0-0266-482a-a845-6b1098e1bd53 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:49:29.565368 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:49:29.575542 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-37d2b1f0-0266-482a-a845-6b1098e1bd53 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '26e324399ad64863a994246cc04a4751', 'request_id': 'req-37d2b1f0-0266-482a-a845-6b1098e1bd53'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:49:29.585555 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-37d2b1f0-0266-482a-a845-6b1098e1bd53 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '26e324399ad64863a994246cc04a4751', 'request_id': 'req-37d2b1f0-0266-482a-a845-6b1098e1bd53'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:49:29.599319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-37d2b1f0-0266-482a-a845-6b1098e1bd53 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '26e324399ad64863a994246cc04a4751', 'request_id': 'req-37d2b1f0-0266-482a-a845-6b1098e1bd53'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:49:29.615702 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-37d2b1f0-0266-482a-a845-6b1098e1bd53 admin admin] The token cache is being invalidate because project 26e324399ad64863a994246cc04a4751 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:49:29.616058 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-37d2b1f0-0266-482a-a845-6b1098e1bd53 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:49:29.617529 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-37d2b1f0-0266-482a-a845-6b1098e1bd53 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:49:29.632961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1102/2179] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:49:29 2022] DELETE /identity/v3/projects/26e324399ad64863a994246cc04a4751 => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:49:29.633854 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd47bb7-f477-4150-8041-0a4b7acb577f None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:29.634166 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd47bb7-f477-4150-8041-0a4b7acb577f None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:29.635079 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd47bb7-f477-4150-8041-0a4b7acb577f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:29.671300 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-7cd47bb7-f477-4150-8041-0a4b7acb577f None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:49:29.674571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-7cd47bb7-f477-4150-8041-0a4b7acb577f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:49:29.751754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:49:29.838942 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1078/2180] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:49:29 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:49:29.846301 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b65b2a4-70cf-496f-b571-a5aaef781a4b None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:29.846598 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b65b2a4-70cf-496f-b571-a5aaef781a4b None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:29.846856 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b65b2a4-70cf-496f-b571-a5aaef781a4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:29.880226 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-7b65b2a4-70cf-496f-b571-a5aaef781a4b None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:49:29.883192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-7b65b2a4-70cf-496f-b571-a5aaef781a4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:49:29.892430 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1103/2181] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:49:29 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:49:29.897047 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd47bb7-f477-4150-8041-0a4b7acb577f None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:49:29.912547 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd47bb7-f477-4150-8041-0a4b7acb577f None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:49:29.924590 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a806baa-201b-4ee2-8a46-7f4066c55b96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:49:29.927196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a806baa-201b-4ee2-8a46-7f4066c55b96 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:29.927743 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a806baa-201b-4ee2-8a46-7f4066c55b96 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:29.928242 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a806baa-201b-4ee2-8a46-7f4066c55b96 admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:29.929114 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a806baa-201b-4ee2-8a46-7f4066c55b96 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:49:29.929418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a806baa-201b-4ee2-8a46-7f4066c55b96 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:49:29.931807 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a806baa-201b-4ee2-8a46-7f4066c55b96 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:49:29.948022 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1079/2182] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:49:29 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:49:29.952453 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b65b2a4-70cf-496f-b571-a5aaef781a4b None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:49:29.968570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b65b2a4-70cf-496f-b571-a5aaef781a4b None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:49:29.973791 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed966d8-1651-4c07-a6fb-a8f3ddea245f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:49:29.974690 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed966d8-1651-4c07-a6fb-a8f3ddea245f admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:29.974840 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed966d8-1651-4c07-a6fb-a8f3ddea245f admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:29.975171 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed966d8-1651-4c07-a6fb-a8f3ddea245f admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:29.976154 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed966d8-1651-4c07-a6fb-a8f3ddea245f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:49:29.981414 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed966d8-1651-4c07-a6fb-a8f3ddea245f admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:49:30.018410 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1104/2183] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:49:29 2022] POST /identity/v3/projects => generated 374 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:49:30.022969 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a806baa-201b-4ee2-8a46-7f4066c55b96 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:49:30.034658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a806baa-201b-4ee2-8a46-7f4066c55b96 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:49:30.111819 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d5f7ff-b2d6-463a-b882-6c3fe3ba338d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:49:30.112595 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d5f7ff-b2d6-463a-b882-6c3fe3ba338d admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:30.112933 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d5f7ff-b2d6-463a-b882-6c3fe3ba338d admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:30.113208 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d5f7ff-b2d6-463a-b882-6c3fe3ba338d admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:30.113842 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d5f7ff-b2d6-463a-b882-6c3fe3ba338d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:49:30.115804 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d5f7ff-b2d6-463a-b882-6c3fe3ba338d admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:49:30.153758 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1080/2184] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:49:30 2022] POST /identity/v3/users => generated 288 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 15:49:30.156983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed966d8-1651-4c07-a6fb-a8f3ddea245f admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:49:30.170950 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed966d8-1651-4c07-a6fb-a8f3ddea245f admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:49:30.177658 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbb280c-ce9b-4fb3-bda2-a7f0b69dcfd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:49:30.178264 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbb280c-ce9b-4fb3-bda2-a7f0b69dcfd3 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:30.178547 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbb280c-ce9b-4fb3-bda2-a7f0b69dcfd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:30.178819 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbb280c-ce9b-4fb3-bda2-a7f0b69dcfd3 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:30.179484 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbb280c-ce9b-4fb3-bda2-a7f0b69dcfd3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:49:30.181805 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbb280c-ce9b-4fb3-bda2-a7f0b69dcfd3 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:49:30.204814 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1105/2185] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:49:30 2022] GET /identity/v3/roles => generated 1427 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:49:30.207103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d5f7ff-b2d6-463a-b882-6c3fe3ba338d admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:49:30.220736 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d5f7ff-b2d6-463a-b882-6c3fe3ba338d admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:49:30.226846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901338ae-6ba3-4ac7-ba18-6fcade01095b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:49:30.227697 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901338ae-6ba3-4ac7-ba18-6fcade01095b admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:30.228506 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901338ae-6ba3-4ac7-ba18-6fcade01095b admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:30.228921 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901338ae-6ba3-4ac7-ba18-6fcade01095b admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:30.229883 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901338ae-6ba3-4ac7-ba18-6fcade01095b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:49:30.231492 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901338ae-6ba3-4ac7-ba18-6fcade01095b admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:49:30.248949 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1081/2186] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:49:30 2022] GET /identity/v3/roles => generated 1427 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:49:30.250442 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbb280c-ce9b-4fb3-bda2-a7f0b69dcfd3 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:49:30.264750 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbb280c-ce9b-4fb3-bda2-a7f0b69dcfd3 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:49:30.275160 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1123a63-e1ff-4bbb-a766-6cef4e917ae1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:49:30.276487 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1123a63-e1ff-4bbb-a766-6cef4e917ae1 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:30.276779 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1123a63-e1ff-4bbb-a766-6cef4e917ae1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:30.277064 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1123a63-e1ff-4bbb-a766-6cef4e917ae1 admin admin] PATH_INFO: `/v3/projects/284bb8dd87ec4ad6819b97181477dce3/users/ad521a5514614e87a2f67c7462f1cf35/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:30.302326 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1123a63-e1ff-4bbb-a766-6cef4e917ae1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=284bb8dd87ec4ad6819b97181477dce3, user_id=ad521a5514614e87a2f67c7462f1cf35, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:49:30.305098 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1123a63-e1ff-4bbb-a766-6cef4e917ae1 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:49:30.328794 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1106/2187] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:49:30 2022] PUT /identity/v3/projects/284bb8dd87ec4ad6819b97181477dce3/users/ad521a5514614e87a2f67c7462f1cf35/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:49:30.340691 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee71056-b138-49ca-8a60-50bd2e040a38 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:30.341199 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee71056-b138-49ca-8a60-50bd2e040a38 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:30.341293 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee71056-b138-49ca-8a60-50bd2e040a38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:30.379377 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-3ee71056-b138-49ca-8a60-50bd2e040a38 None None] MFA Rules not processed for user `ad521a5514614e87a2f67c7462f1cf35`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:49:30.381779 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-3ee71056-b138-49ca-8a60-50bd2e040a38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:49:30.480228 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1082/2188] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:49:30 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:49:30.487255 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1123a63-e1ff-4bbb-a766-6cef4e917ae1 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:49:30.489236 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-a1123a63-e1ff-4bbb-a766-6cef4e917ae1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:49:30.584713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1123a63-e1ff-4bbb-a766-6cef4e917ae1 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:49:30.590345 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9738c372-6d3e-46e3-97e7-862fbb861c2e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:49:30.590970 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9738c372-6d3e-46e3-97e7-862fbb861c2e service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:30.591338 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9738c372-6d3e-46e3-97e7-862fbb861c2e service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:30.591733 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9738c372-6d3e-46e3-97e7-862fbb861c2e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:30.600649 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9738c372-6d3e-46e3-97e7-862fbb861c2e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:49:30.602256 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9738c372-6d3e-46e3-97e7-862fbb861c2e service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:49:30.616823 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1107/2189] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:49:30 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 15:49:32.245955 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b41a08-1c81-47ad-9d78-0f5eb3074451 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:32.246529 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b41a08-1c81-47ad-9d78-0f5eb3074451 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:32.246529 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b41a08-1c81-47ad-9d78-0f5eb3074451 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:32.282767 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-d6b41a08-1c81-47ad-9d78-0f5eb3074451 None None] MFA Rules not processed for user `ad521a5514614e87a2f67c7462f1cf35`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:49:32.286460 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-d6b41a08-1c81-47ad-9d78-0f5eb3074451 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:49:32.298139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1083/2190] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:49:32 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:49:35.496210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cfebdc-cb82-4cbf-9ccd-838c8531a816 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:35.496759 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cfebdc-cb82-4cbf-9ccd-838c8531a816 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:35.496759 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cfebdc-cb82-4cbf-9ccd-838c8531a816 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:35.521466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-c0cfebdc-cb82-4cbf-9ccd-838c8531a816 None None] MFA Rules not processed for user `2fe5724620da46ddbefbdfd442dc8d73`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:49:35.522983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-c0cfebdc-cb82-4cbf-9ccd-838c8531a816 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:49:35.594787 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1108/2191] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:49:35 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:49:42.270658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=94,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:49:44.561588 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798b1617-4dde-4d98-b594-9f7c1c7b83cd None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:44.562127 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798b1617-4dde-4d98-b594-9f7c1c7b83cd None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:44.562127 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798b1617-4dde-4d98-b594-9f7c1c7b83cd None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:44.563977 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1084/2192] 10.222.0.102 () {54 vars in 926 bytes} [Wed Aug 24 15:49:44 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:49:44.570448 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc70642b-a3ed-47c2-981b-372431932458 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:44.570981 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc70642b-a3ed-47c2-981b-372431932458 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:44.571249 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc70642b-a3ed-47c2-981b-372431932458 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:44.599579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-fc70642b-a3ed-47c2-981b-372431932458 None None] MFA Rules not processed for user `e0d544379fe44438ad10289b45124374`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:49:44.600904 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-fc70642b-a3ed-47c2-981b-372431932458 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:49:44.628627 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:49:44.734934 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1109/2193] 10.222.0.102 () {58 vars in 1023 bytes} [Wed Aug 24 15:49:44 2022] POST /identity/v3/auth/tokens => generated 3382 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:49:44.749389 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798b1617-4dde-4d98-b594-9f7c1c7b83cd None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:49:44.751185 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-798b1617-4dde-4d98-b594-9f7c1c7b83cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:49:44.864929 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798b1617-4dde-4d98-b594-9f7c1c7b83cd None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:49:44.872161 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851bb685-55c4-482b-b550-fadd7cdfd69f service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c6f06186c1a4626997db5a40c31cbb9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:49:44.872777 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851bb685-55c4-482b-b550-fadd7cdfd69f service glance] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:44.873059 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851bb685-55c4-482b-b550-fadd7cdfd69f service glance] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:44.873332 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851bb685-55c4-482b-b550-fadd7cdfd69f service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:44.897229 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851bb685-55c4-482b-b550-fadd7cdfd69f service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:49:44.899897 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851bb685-55c4-482b-b550-fadd7cdfd69f service glance] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:49:44.925767 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1085/2194] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:49:44 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:49:45.021802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92de278c-2721-4591-9488-4a76d2534218 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:45.022100 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92de278c-2721-4591-9488-4a76d2534218 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:45.022379 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92de278c-2721-4591-9488-4a76d2534218 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:45.023780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1110/2195] 10.222.0.102 () {54 vars in 926 bytes} [Wed Aug 24 15:49:45 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:49:45.032296 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f97ec6-a581-40c7-bea6-08030e068f2e None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:45.032680 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f97ec6-a581-40c7-bea6-08030e068f2e None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:45.033322 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f97ec6-a581-40c7-bea6-08030e068f2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:45.056412 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-89f97ec6-a581-40c7-bea6-08030e068f2e None None] MFA Rules not processed for user `48939674c1fd4cf599691d510fcd8858`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:49:45.057717 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-89f97ec6-a581-40c7-bea6-08030e068f2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:49:45.113653 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1086/2196] 10.222.0.102 () {58 vars in 1023 bytes} [Wed Aug 24 15:49:45 2022] POST /identity/v3/auth/tokens => generated 3131 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:49:45.125806 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92de278c-2721-4591-9488-4a76d2534218 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:49:45.127890 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-92de278c-2721-4591-9488-4a76d2534218 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:49:45.141618 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92de278c-2721-4591-9488-4a76d2534218 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:49:45.145263 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8c03bc-1d9a-43a2-9b42-faca1409e55b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:49:45.145861 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8c03bc-1d9a-43a2-9b42-faca1409e55b service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:45.146144 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8c03bc-1d9a-43a2-9b42-faca1409e55b service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:45.146420 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8c03bc-1d9a-43a2-9b42-faca1409e55b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:45.155513 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8c03bc-1d9a-43a2-9b42-faca1409e55b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:49:45.157031 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8c03bc-1d9a-43a2-9b42-faca1409e55b service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:49:45.165519 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1111/2197] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:49:45 2022] GET /identity/v3/auth/tokens => generated 3131 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:49:45.268599 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094fa382-2b70-4db4-858f-fd17e3932b80 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:45.268942 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094fa382-2b70-4db4-858f-fd17e3932b80 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:45.269213 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094fa382-2b70-4db4-858f-fd17e3932b80 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:45.270685 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1087/2198] 10.222.0.102 () {54 vars in 926 bytes} [Wed Aug 24 15:49:45 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 15:49:45.277787 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6f4418-c575-4622-a34c-f078b7304d41 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:45.278420 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6f4418-c575-4622-a34c-f078b7304d41 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:45.278588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6f4418-c575-4622-a34c-f078b7304d41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:45.301316 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-9c6f4418-c575-4622-a34c-f078b7304d41 None None] MFA Rules not processed for user `e0d544379fe44438ad10289b45124374`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:49:45.303592 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-9c6f4418-c575-4622-a34c-f078b7304d41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:49:45.315444 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1112/2199] 10.222.0.102 () {58 vars in 1023 bytes} [Wed Aug 24 15:49:45 2022] POST /identity/v3/auth/tokens => generated 3382 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:49:45.327328 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-094fa382-2b70-4db4-858f-fd17e3932b80 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:49:45.340224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-094fa382-2b70-4db4-858f-fd17e3932b80 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:49:45.344012 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6baa4655-0f40-4f75-9471-8d6f8cbec905 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:49:45.344611 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6baa4655-0f40-4f75-9471-8d6f8cbec905 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:45.344893 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6baa4655-0f40-4f75-9471-8d6f8cbec905 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:45.345170 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6baa4655-0f40-4f75-9471-8d6f8cbec905 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:45.356111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6baa4655-0f40-4f75-9471-8d6f8cbec905 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:49:45.357891 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6baa4655-0f40-4f75-9471-8d6f8cbec905 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:49:45.369323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1088/2200] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:49:45 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:49:55.294214 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=34 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:49:55.364445 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=55 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:49:57.358292 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6f4418-c575-4622-a34c-f078b7304d41 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:49:57.359895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-9c6f4418-c575-4622-a34c-f078b7304d41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:49:57.429095 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6f4418-c575-4622-a34c-f078b7304d41 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:49:57.434728 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679c7991-0448-4da3-ae17-d2c9549a644a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:49:57.435284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679c7991-0448-4da3-ae17-d2c9549a644a admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:57.435570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679c7991-0448-4da3-ae17-d2c9549a644a admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:57.435879 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679c7991-0448-4da3-ae17-d2c9549a644a admin admin] PATH_INFO: `/v3/users/75117fe6092347c58cd3384c06eea72b` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:57.437050 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679c7991-0448-4da3-ae17-d2c9549a644a admin admin] RBAC: Authorizing `identity:delete_user(user_id=75117fe6092347c58cd3384c06eea72b)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:49:57.438985 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679c7991-0448-4da3-ae17-d2c9549a644a admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:49:57.493173 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-679c7991-0448-4da3-ae17-d2c9549a644a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '75117fe6092347c58cd3384c06eea72b'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:49:57.493832 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-679c7991-0448-4da3-ae17-d2c9549a644a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '75117fe6092347c58cd3384c06eea72b'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:49:57.494439 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-679c7991-0448-4da3-ae17-d2c9549a644a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '75117fe6092347c58cd3384c06eea72b'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:49:57.501715 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-679c7991-0448-4da3-ae17-d2c9549a644a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '75117fe6092347c58cd3384c06eea72b'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:49:57.514601 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-679c7991-0448-4da3-ae17-d2c9549a644a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '75117fe6092347c58cd3384c06eea72b'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:49:57.528971 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1113/2201] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:49:57 2022] DELETE /identity/v3/users/75117fe6092347c58cd3384c06eea72b => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 15:49:57.533509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2aee5f7-7335-439b-a36c-e8a6e69d884b None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:57.533988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2aee5f7-7335-439b-a36c-e8a6e69d884b None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:57.534108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2aee5f7-7335-439b-a36c-e8a6e69d884b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:57.552869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-f2aee5f7-7335-439b-a36c-e8a6e69d884b None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:49:57.554078 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-f2aee5f7-7335-439b-a36c-e8a6e69d884b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:49:57.617354 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1089/2202] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:49:57 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:49:57.627720 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679c7991-0448-4da3-ae17-d2c9549a644a admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:49:57.629278 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-679c7991-0448-4da3-ae17-d2c9549a644a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:49:57.690955 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679c7991-0448-4da3-ae17-d2c9549a644a admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:49:57.694767 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad48ab8-7788-4c4a-a71b-bca85ea4ea41 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:49:57.695366 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad48ab8-7788-4c4a-a71b-bca85ea4ea41 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:57.695667 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad48ab8-7788-4c4a-a71b-bca85ea4ea41 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:57.695950 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad48ab8-7788-4c4a-a71b-bca85ea4ea41 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:57.708808 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ad48ab8-7788-4c4a-a71b-bca85ea4ea41 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:49:57.710730 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ad48ab8-7788-4c4a-a71b-bca85ea4ea41 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:49:57.724173 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1114/2203] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:49:57 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:49:58.162887 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2aee5f7-7335-439b-a36c-e8a6e69d884b None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:49:58.164337 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-f2aee5f7-7335-439b-a36c-e8a6e69d884b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:49:58.176240 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2aee5f7-7335-439b-a36c-e8a6e69d884b None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:49:58.181538 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c9f0c2-439f-432f-84e7-61004e306265 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:49:58.182228 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c9f0c2-439f-432f-84e7-61004e306265 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:49:58.182674 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c9f0c2-439f-432f-84e7-61004e306265 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:49:58.182949 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c9f0c2-439f-432f-84e7-61004e306265 admin admin] PATH_INFO: `/v3/projects/bd10d33165a345179b73fc117e1c084c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:49:58.184117 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c9f0c2-439f-432f-84e7-61004e306265 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bd10d33165a345179b73fc117e1c084c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:49:58.186019 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c9f0c2-439f-432f-84e7-61004e306265 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:49:58.241645 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-a0c9f0c2-439f-432f-84e7-61004e306265 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bd10d33165a345179b73fc117e1c084c', 'request_id': 'req-a0c9f0c2-439f-432f-84e7-61004e306265'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:49:58.248723 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-a0c9f0c2-439f-432f-84e7-61004e306265 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bd10d33165a345179b73fc117e1c084c', 'request_id': 'req-a0c9f0c2-439f-432f-84e7-61004e306265'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:49:58.261856 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-a0c9f0c2-439f-432f-84e7-61004e306265 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bd10d33165a345179b73fc117e1c084c', 'request_id': 'req-a0c9f0c2-439f-432f-84e7-61004e306265'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:49:58.282960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-a0c9f0c2-439f-432f-84e7-61004e306265 admin admin] The token cache is being invalidate because project bd10d33165a345179b73fc117e1c084c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:49:58.283258 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-a0c9f0c2-439f-432f-84e7-61004e306265 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:49:58.283937 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-a0c9f0c2-439f-432f-84e7-61004e306265 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:49:58.285619 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1090/2204] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:49:58 2022] DELETE /identity/v3/projects/bd10d33165a345179b73fc117e1c084c => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:50:03.858403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad48ab8-7788-4c4a-a71b-bca85ea4ea41 service neutron] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:03.860196 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-2ad48ab8-7788-4c4a-a71b-bca85ea4ea41 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:50:03.890923 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:50:03.942295 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad48ab8-7788-4c4a-a71b-bca85ea4ea41 service neutron] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:04.028744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffebf2b-e3bf-4db8-b5ec-3ed5af833944 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0ee0002a76c3471f819b37fef510ef99', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:04.029363 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffebf2b-e3bf-4db8-b5ec-3ed5af833944 service placement] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:04.029647 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffebf2b-e3bf-4db8-b5ec-3ed5af833944 service placement] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:04.029922 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffebf2b-e3bf-4db8-b5ec-3ed5af833944 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:04.031795 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-6ffebf2b-e3bf-4db8-b5ec-3ed5af833944 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:50:04.044045 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffebf2b-e3bf-4db8-b5ec-3ed5af833944 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:04.045643 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffebf2b-e3bf-4db8-b5ec-3ed5af833944 service placement] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:04.055979 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1115/2205] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:50:03 2022] GET /identity/v3/auth/tokens => generated 3318 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:50:08.279464 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=54,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:50:14.051749 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=48 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:50:21.731615 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae66fb81-d259-4dcc-b194-15a75d811f35 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:21.732114 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae66fb81-d259-4dcc-b194-15a75d811f35 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:21.732480 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae66fb81-d259-4dcc-b194-15a75d811f35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:21.753725 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-ae66fb81-d259-4dcc-b194-15a75d811f35 None None] MFA Rules not processed for user `a6c17620ead14cb49d962990e1b1e0d8`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:50:21.755053 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-ae66fb81-d259-4dcc-b194-15a75d811f35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:50:21.828127 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1091/2206] 10.222.0.102 () {58 vars in 1014 bytes} [Wed Aug 24 15:50:21 2022] POST /identity/v3/auth/tokens => generated 3205 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:50:21.852054 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffebf2b-e3bf-4db8-b5ec-3ed5af833944 service placement] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:21.853538 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-6ffebf2b-e3bf-4db8-b5ec-3ed5af833944 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:50:21.949146 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffebf2b-e3bf-4db8-b5ec-3ed5af833944 service placement] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:21.956242 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250497bf-768b-4f7d-bf41-154ba065caf7 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a070d4fd8c4466c857bd03965c11349', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:21.956947 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250497bf-768b-4f7d-bf41-154ba065caf7 service swift] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:21.957245 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250497bf-768b-4f7d-bf41-154ba065caf7 service swift] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:21.957521 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250497bf-768b-4f7d-bf41-154ba065caf7 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:21.966625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250497bf-768b-4f7d-bf41-154ba065caf7 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:21.968718 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250497bf-768b-4f7d-bf41-154ba065caf7 service swift] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:21.977912 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1116/2207] 10.222.0.102 () {60 vars in 1389 bytes} [Wed Aug 24 15:50:21 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 24 15:50:22.750389 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16b74be-401e-4039-b964-32cf08b859c3 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:22.750904 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16b74be-401e-4039-b964-32cf08b859c3 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:22.751026 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16b74be-401e-4039-b964-32cf08b859c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:22.771528 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-d16b74be-401e-4039-b964-32cf08b859c3 None None] MFA Rules not processed for user `a6c17620ead14cb49d962990e1b1e0d8`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:50:22.772813 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-d16b74be-401e-4039-b964-32cf08b859c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:50:22.781219 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1092/2208] 10.222.0.102 () {58 vars in 1014 bytes} [Wed Aug 24 15:50:22 2022] POST /identity/v3/auth/tokens => generated 3205 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:50:22.793859 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250497bf-768b-4f7d-bf41-154ba065caf7 service swift] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:22.804920 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250497bf-768b-4f7d-bf41-154ba065caf7 service swift] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:22.810763 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1456bfa6-0e0b-4248-a64e-c69558538798 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a070d4fd8c4466c857bd03965c11349', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:22.811492 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1456bfa6-0e0b-4248-a64e-c69558538798 service swift] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:22.811808 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1456bfa6-0e0b-4248-a64e-c69558538798 service swift] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:22.812085 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1456bfa6-0e0b-4248-a64e-c69558538798 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:22.822638 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1456bfa6-0e0b-4248-a64e-c69558538798 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:22.824168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1456bfa6-0e0b-4248-a64e-c69558538798 service swift] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:22.835133 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1117/2209] 10.222.0.102 () {60 vars in 1389 bytes} [Wed Aug 24 15:50:22 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 24 15:50:23.458187 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14aac0c-bf30-41a3-9549-c0c5cfe7c33d None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:23.458404 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14aac0c-bf30-41a3-9549-c0c5cfe7c33d None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:23.458701 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14aac0c-bf30-41a3-9549-c0c5cfe7c33d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:23.478847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-f14aac0c-bf30-41a3-9549-c0c5cfe7c33d None None] MFA Rules not processed for user `a6c17620ead14cb49d962990e1b1e0d8`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:50:23.480169 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-f14aac0c-bf30-41a3-9549-c0c5cfe7c33d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:50:23.488447 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1093/2210] 10.222.0.102 () {58 vars in 972 bytes} [Wed Aug 24 15:50:23 2022] POST /identity/v3/auth/tokens => generated 3205 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:50:23.511875 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1456bfa6-0e0b-4248-a64e-c69558538798 service swift] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:23.525174 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1456bfa6-0e0b-4248-a64e-c69558538798 service swift] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:23.531277 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fe08be-e293-4721-8096-0ccdf8c27517 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a070d4fd8c4466c857bd03965c11349', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:23.531967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fe08be-e293-4721-8096-0ccdf8c27517 service swift] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:23.532262 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fe08be-e293-4721-8096-0ccdf8c27517 service swift] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:23.532540 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fe08be-e293-4721-8096-0ccdf8c27517 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:23.541637 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89fe08be-e293-4721-8096-0ccdf8c27517 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:23.543372 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89fe08be-e293-4721-8096-0ccdf8c27517 service swift] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:23.553658 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1118/2211] 10.222.0.102 () {60 vars in 1389 bytes} [Wed Aug 24 15:50:23 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 24 15:50:23.795985 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f516a036-7b39-4889-aee3-d632f2f262b1 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:23.796442 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f516a036-7b39-4889-aee3-d632f2f262b1 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:23.796530 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f516a036-7b39-4889-aee3-d632f2f262b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:23.816127 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-f516a036-7b39-4889-aee3-d632f2f262b1 None None] MFA Rules not processed for user `a6c17620ead14cb49d962990e1b1e0d8`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:50:23.817318 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-f516a036-7b39-4889-aee3-d632f2f262b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:50:23.825649 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1094/2212] 10.222.0.102 () {58 vars in 1014 bytes} [Wed Aug 24 15:50:23 2022] POST /identity/v3/auth/tokens => generated 3205 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:50:23.838305 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fe08be-e293-4721-8096-0ccdf8c27517 service swift] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:23.856046 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fe08be-e293-4721-8096-0ccdf8c27517 service swift] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:23.864402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e887ca-801b-4f00-8339-fd5bf481cfbd service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a070d4fd8c4466c857bd03965c11349', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:23.865124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e887ca-801b-4f00-8339-fd5bf481cfbd service swift] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:23.865414 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e887ca-801b-4f00-8339-fd5bf481cfbd service swift] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:23.865695 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e887ca-801b-4f00-8339-fd5bf481cfbd service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:23.876236 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e887ca-801b-4f00-8339-fd5bf481cfbd service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:23.877956 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e887ca-801b-4f00-8339-fd5bf481cfbd service swift] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:23.888356 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1119/2213] 10.222.0.102 () {60 vars in 1389 bytes} [Wed Aug 24 15:50:23 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 24 15:50:24.240289 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987a1b43-16dd-4ba3-aaf8-678716eb5b5a None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:24.240578 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987a1b43-16dd-4ba3-aaf8-678716eb5b5a None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:24.240863 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987a1b43-16dd-4ba3-aaf8-678716eb5b5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:24.264856 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-987a1b43-16dd-4ba3-aaf8-678716eb5b5a None None] MFA Rules not processed for user `a6c17620ead14cb49d962990e1b1e0d8`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:50:24.266247 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-987a1b43-16dd-4ba3-aaf8-678716eb5b5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:50:24.274448 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1095/2214] 10.222.0.102 () {58 vars in 972 bytes} [Wed Aug 24 15:50:24 2022] POST /identity/v3/auth/tokens => generated 3205 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:50:24.303535 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e887ca-801b-4f00-8339-fd5bf481cfbd service swift] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:24.317192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e887ca-801b-4f00-8339-fd5bf481cfbd service swift] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:24.329273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a8d05f-4252-4797-aaee-5f332d9c0cdd service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a070d4fd8c4466c857bd03965c11349', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:24.329273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a8d05f-4252-4797-aaee-5f332d9c0cdd service swift] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:24.329273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a8d05f-4252-4797-aaee-5f332d9c0cdd service swift] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:24.329273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a8d05f-4252-4797-aaee-5f332d9c0cdd service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:24.336543 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a8d05f-4252-4797-aaee-5f332d9c0cdd service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:24.338244 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a8d05f-4252-4797-aaee-5f332d9c0cdd service swift] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:24.348161 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1120/2215] 10.222.0.102 () {60 vars in 1389 bytes} [Wed Aug 24 15:50:24 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 24 15:50:26.631205 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987a1b43-16dd-4ba3-aaf8-678716eb5b5a None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:26.633472 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-987a1b43-16dd-4ba3-aaf8-678716eb5b5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:50:26.708533 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987a1b43-16dd-4ba3-aaf8-678716eb5b5a None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:26.713123 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07985c2-b3bc-45ed-af38-161377e158e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:26.714049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07985c2-b3bc-45ed-af38-161377e158e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:26.714580 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07985c2-b3bc-45ed-af38-161377e158e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:26.715042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07985c2-b3bc-45ed-af38-161377e158e0 admin admin] PATH_INFO: `/v3/users/ad521a5514614e87a2f67c7462f1cf35` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:26.717945 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07985c2-b3bc-45ed-af38-161377e158e0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ad521a5514614e87a2f67c7462f1cf35)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:26.720133 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07985c2-b3bc-45ed-af38-161377e158e0 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:26.791385 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-a07985c2-b3bc-45ed-af38-161377e158e0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ad521a5514614e87a2f67c7462f1cf35'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:50:26.792030 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-a07985c2-b3bc-45ed-af38-161377e158e0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ad521a5514614e87a2f67c7462f1cf35'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:50:26.810251 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-a07985c2-b3bc-45ed-af38-161377e158e0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ad521a5514614e87a2f67c7462f1cf35'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:50:26.846769 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-a07985c2-b3bc-45ed-af38-161377e158e0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ad521a5514614e87a2f67c7462f1cf35'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:50:26.848137 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-a07985c2-b3bc-45ed-af38-161377e158e0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ad521a5514614e87a2f67c7462f1cf35'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:50:26.870487 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:50:26.896802 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1096/2216] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:50:26 2022] DELETE /identity/v3/users/ad521a5514614e87a2f67c7462f1cf35 => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:50:26.896902 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74727575-218b-46b7-9971-be5f035be4a5 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:26.896902 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74727575-218b-46b7-9971-be5f035be4a5 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:26.896902 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74727575-218b-46b7-9971-be5f035be4a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:26.923963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-74727575-218b-46b7-9971-be5f035be4a5 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:50:26.925538 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-74727575-218b-46b7-9971-be5f035be4a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:50:27.006916 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1121/2217] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:50:26 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:50:27.017479 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07985c2-b3bc-45ed-af38-161377e158e0 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:27.019007 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-a07985c2-b3bc-45ed-af38-161377e158e0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:50:27.082367 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07985c2-b3bc-45ed-af38-161377e158e0 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:27.086052 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ff0272-464a-4cea-aa4a-c941eb2eb4af service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:27.086648 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ff0272-464a-4cea-aa4a-c941eb2eb4af service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:27.086929 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ff0272-464a-4cea-aa4a-c941eb2eb4af service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:27.087203 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ff0272-464a-4cea-aa4a-c941eb2eb4af service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:27.096668 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ff0272-464a-4cea-aa4a-c941eb2eb4af service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:27.098108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ff0272-464a-4cea-aa4a-c941eb2eb4af service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:27.110434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1097/2218] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:50:27 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:50:27.304542 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74727575-218b-46b7-9971-be5f035be4a5 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:27.306402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-74727575-218b-46b7-9971-be5f035be4a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:50:27.328127 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74727575-218b-46b7-9971-be5f035be4a5 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:27.343658 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd39f80-530a-462c-9c33-a056e04fc1f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:27.344370 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd39f80-530a-462c-9c33-a056e04fc1f9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:27.344654 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd39f80-530a-462c-9c33-a056e04fc1f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:27.344932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd39f80-530a-462c-9c33-a056e04fc1f9 admin admin] PATH_INFO: `/v3/projects/284bb8dd87ec4ad6819b97181477dce3` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:27.346190 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd39f80-530a-462c-9c33-a056e04fc1f9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=284bb8dd87ec4ad6819b97181477dce3)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:27.348222 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd39f80-530a-462c-9c33-a056e04fc1f9 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:27.384711 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:50:27.407013 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0fd39f80-530a-462c-9c33-a056e04fc1f9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '284bb8dd87ec4ad6819b97181477dce3', 'request_id': 'req-0fd39f80-530a-462c-9c33-a056e04fc1f9'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:50:27.415667 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0fd39f80-530a-462c-9c33-a056e04fc1f9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '284bb8dd87ec4ad6819b97181477dce3', 'request_id': 'req-0fd39f80-530a-462c-9c33-a056e04fc1f9'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:50:27.422831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0fd39f80-530a-462c-9c33-a056e04fc1f9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '284bb8dd87ec4ad6819b97181477dce3', 'request_id': 'req-0fd39f80-530a-462c-9c33-a056e04fc1f9'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:50:27.438362 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0fd39f80-530a-462c-9c33-a056e04fc1f9 admin admin] The token cache is being invalidate because project 284bb8dd87ec4ad6819b97181477dce3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:50:27.438658 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0fd39f80-530a-462c-9c33-a056e04fc1f9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:50:27.439362 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0fd39f80-530a-462c-9c33-a056e04fc1f9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:50:27.440919 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1122/2219] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:50:27 2022] DELETE /identity/v3/projects/284bb8dd87ec4ad6819b97181477dce3 => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:50:27.445823 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6c2d14-e712-46b5-9f51-831b91164b04 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:27.446109 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6c2d14-e712-46b5-9f51-831b91164b04 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:27.446386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6c2d14-e712-46b5-9f51-831b91164b04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:27.467378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-ff6c2d14-e712-46b5-9f51-831b91164b04 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:50:27.468420 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-ff6c2d14-e712-46b5-9f51-831b91164b04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:50:27.549119 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1098/2220] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:50:27 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:50:27.557627 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a839233e-1b23-41ca-aad0-4409d5f367bf None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:27.558061 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a839233e-1b23-41ca-aad0-4409d5f367bf None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:27.558251 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a839233e-1b23-41ca-aad0-4409d5f367bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:27.581655 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-a839233e-1b23-41ca-aad0-4409d5f367bf None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:50:27.582856 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-a839233e-1b23-41ca-aad0-4409d5f367bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:50:27.590804 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1123/2221] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:50:27 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:50:27.595163 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6c2d14-e712-46b5-9f51-831b91164b04 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:27.623778 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6c2d14-e712-46b5-9f51-831b91164b04 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:27.639766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38783d22-f3d6-43ab-ad33-4fd601cd6d71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:27.641747 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38783d22-f3d6-43ab-ad33-4fd601cd6d71 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:27.642482 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38783d22-f3d6-43ab-ad33-4fd601cd6d71 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:27.643264 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38783d22-f3d6-43ab-ad33-4fd601cd6d71 admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:27.644393 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38783d22-f3d6-43ab-ad33-4fd601cd6d71 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:50:27.644806 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38783d22-f3d6-43ab-ad33-4fd601cd6d71 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:27.646803 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38783d22-f3d6-43ab-ad33-4fd601cd6d71 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:27.665937 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1099/2222] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:50:27 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 15:50:27.677510 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a839233e-1b23-41ca-aad0-4409d5f367bf None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:27.692921 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a839233e-1b23-41ca-aad0-4409d5f367bf None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:27.701266 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54eb76c2-2ca5-4880-abcb-aa99567657bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:27.702056 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54eb76c2-2ca5-4880-abcb-aa99567657bc admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:27.702440 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54eb76c2-2ca5-4880-abcb-aa99567657bc admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:27.702798 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54eb76c2-2ca5-4880-abcb-aa99567657bc admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:27.703604 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54eb76c2-2ca5-4880-abcb-aa99567657bc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:27.706008 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54eb76c2-2ca5-4880-abcb-aa99567657bc admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:27.742369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1124/2223] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:50:27 2022] POST /identity/v3/projects => generated 400 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:50:27.746564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38783d22-f3d6-43ab-ad33-4fd601cd6d71 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:27.758306 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38783d22-f3d6-43ab-ad33-4fd601cd6d71 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:27.836159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa46889-737a-4638-8025-a18f9912fc7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:27.836775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa46889-737a-4638-8025-a18f9912fc7c admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:27.837060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa46889-737a-4638-8025-a18f9912fc7c admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:27.837338 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa46889-737a-4638-8025-a18f9912fc7c admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:27.838002 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa46889-737a-4638-8025-a18f9912fc7c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:27.840224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa46889-737a-4638-8025-a18f9912fc7c admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:27.868130 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1100/2224] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:50:27 2022] POST /identity/v3/users => generated 301 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 15:50:27.874008 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54eb76c2-2ca5-4880-abcb-aa99567657bc admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:27.885023 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54eb76c2-2ca5-4880-abcb-aa99567657bc admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:27.900434 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84d4b78-7b85-478e-8f87-2b24aa58df53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:27.901538 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84d4b78-7b85-478e-8f87-2b24aa58df53 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:27.902928 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84d4b78-7b85-478e-8f87-2b24aa58df53 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:27.902928 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84d4b78-7b85-478e-8f87-2b24aa58df53 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:27.903546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84d4b78-7b85-478e-8f87-2b24aa58df53 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:27.905855 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84d4b78-7b85-478e-8f87-2b24aa58df53 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:27.920760 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1125/2225] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:50:27 2022] GET /identity/v3/roles => generated 1427 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:50:27.925090 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa46889-737a-4638-8025-a18f9912fc7c admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:27.936833 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa46889-737a-4638-8025-a18f9912fc7c admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:27.941925 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0548dc38-6c92-4832-8261-c974a3d56def admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:27.942697 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0548dc38-6c92-4832-8261-c974a3d56def admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:27.942988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0548dc38-6c92-4832-8261-c974a3d56def admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:27.952227 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0548dc38-6c92-4832-8261-c974a3d56def admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:27.954560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0548dc38-6c92-4832-8261-c974a3d56def admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:27.958821 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0548dc38-6c92-4832-8261-c974a3d56def admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:27.974740 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1101/2226] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:50:27 2022] GET /identity/v3/roles => generated 1427 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:50:27.980918 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84d4b78-7b85-478e-8f87-2b24aa58df53 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:27.993667 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84d4b78-7b85-478e-8f87-2b24aa58df53 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:28.001664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afb79ef-010f-4aa6-a898-5a100fb337cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:28.002444 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afb79ef-010f-4aa6-a898-5a100fb337cb admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:28.002802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afb79ef-010f-4aa6-a898-5a100fb337cb admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:28.003136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afb79ef-010f-4aa6-a898-5a100fb337cb admin admin] PATH_INFO: `/v3/projects/b042d1409b9644aea25bfd4023371265/users/dbc5cdb7819d4918a036e49a3eb2ad22/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:28.021483 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7afb79ef-010f-4aa6-a898-5a100fb337cb admin admin] RBAC: Authorizing `identity:create_grant(project_id=b042d1409b9644aea25bfd4023371265, user_id=dbc5cdb7819d4918a036e49a3eb2ad22, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:28.024354 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7afb79ef-010f-4aa6-a898-5a100fb337cb admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:28.049019 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1126/2227] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:50:27 2022] PUT /identity/v3/projects/b042d1409b9644aea25bfd4023371265/users/dbc5cdb7819d4918a036e49a3eb2ad22/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:50:28.059965 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2287af-c72d-4ae1-8dff-ec4d08a959e1 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:28.060412 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2287af-c72d-4ae1-8dff-ec4d08a959e1 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:28.060694 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2287af-c72d-4ae1-8dff-ec4d08a959e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:28.078758 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:50:28.103166 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-ff2287af-c72d-4ae1-8dff-ec4d08a959e1 None None] MFA Rules not processed for user `dbc5cdb7819d4918a036e49a3eb2ad22`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:50:28.104903 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-ff2287af-c72d-4ae1-8dff-ec4d08a959e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:50:28.207087 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1102/2228] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:50:28 2022] POST /identity/v3/auth/tokens => generated 3286 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:50:28.220071 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afb79ef-010f-4aa6-a898-5a100fb337cb admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:28.221980 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-7afb79ef-010f-4aa6-a898-5a100fb337cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:50:28.284025 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afb79ef-010f-4aa6-a898-5a100fb337cb admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:28.287851 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2b1806-538b-404c-9928-108e079c4e95 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:28.288471 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2b1806-538b-404c-9928-108e079c4e95 service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:28.288754 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2b1806-538b-404c-9928-108e079c4e95 service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:28.289029 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2b1806-538b-404c-9928-108e079c4e95 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:28.299146 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2b1806-538b-404c-9928-108e079c4e95 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:28.300627 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2b1806-538b-404c-9928-108e079c4e95 service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:28.309618 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1127/2229] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:50:28 2022] GET /identity/v3/auth/tokens => generated 3286 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:50:36.486270 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2287af-c72d-4ae1-8dff-ec4d08a959e1 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:36.552299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2287af-c72d-4ae1-8dff-ec4d08a959e1 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:36.556756 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5ca5c4-1690-4dcb-b9a8-837dfbf0616b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:36.557377 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5ca5c4-1690-4dcb-b9a8-837dfbf0616b admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:36.557660 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5ca5c4-1690-4dcb-b9a8-837dfbf0616b admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:36.557958 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5ca5c4-1690-4dcb-b9a8-837dfbf0616b admin admin] PATH_INFO: `/v3/users/dbc5cdb7819d4918a036e49a3eb2ad22` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:36.559093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5ca5c4-1690-4dcb-b9a8-837dfbf0616b admin admin] RBAC: Authorizing `identity:delete_user(user_id=dbc5cdb7819d4918a036e49a3eb2ad22)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:36.562113 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5ca5c4-1690-4dcb-b9a8-837dfbf0616b admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:36.609155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-0d5ca5c4-1690-4dcb-b9a8-837dfbf0616b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dbc5cdb7819d4918a036e49a3eb2ad22'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:50:36.609843 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-0d5ca5c4-1690-4dcb-b9a8-837dfbf0616b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dbc5cdb7819d4918a036e49a3eb2ad22'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:50:36.616764 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-0d5ca5c4-1690-4dcb-b9a8-837dfbf0616b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dbc5cdb7819d4918a036e49a3eb2ad22'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:50:36.629173 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-0d5ca5c4-1690-4dcb-b9a8-837dfbf0616b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dbc5cdb7819d4918a036e49a3eb2ad22'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:50:36.629823 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-0d5ca5c4-1690-4dcb-b9a8-837dfbf0616b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dbc5cdb7819d4918a036e49a3eb2ad22'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:50:36.643425 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1103/2230] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 15:50:36 2022] DELETE /identity/v3/users/dbc5cdb7819d4918a036e49a3eb2ad22 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:50:36.647903 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85de570-4cf5-4ec8-a32f-244d84acacbe None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:36.648436 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85de570-4cf5-4ec8-a32f-244d84acacbe None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:36.648436 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85de570-4cf5-4ec8-a32f-244d84acacbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:36.667955 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-a85de570-4cf5-4ec8-a32f-244d84acacbe None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:50:36.669773 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-a85de570-4cf5-4ec8-a32f-244d84acacbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:50:36.719051 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:50:36.728481 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1128/2231] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:50:36 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:50:36.738475 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5ca5c4-1690-4dcb-b9a8-837dfbf0616b admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:36.740006 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-0d5ca5c4-1690-4dcb-b9a8-837dfbf0616b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:50:36.759580 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:50:36.805421 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5ca5c4-1690-4dcb-b9a8-837dfbf0616b admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:36.810451 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556e4b01-052f-47d9-8316-45391ca541db service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:36.811064 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556e4b01-052f-47d9-8316-45391ca541db service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:36.811342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556e4b01-052f-47d9-8316-45391ca541db service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:36.811611 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556e4b01-052f-47d9-8316-45391ca541db service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:36.826956 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556e4b01-052f-47d9-8316-45391ca541db service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:36.828748 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556e4b01-052f-47d9-8316-45391ca541db service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:36.842465 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1104/2232] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:50:36 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:50:37.138027 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85de570-4cf5-4ec8-a32f-244d84acacbe None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:37.140102 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-a85de570-4cf5-4ec8-a32f-244d84acacbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:50:37.169587 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85de570-4cf5-4ec8-a32f-244d84acacbe None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:37.175392 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08aed8ad-6c06-44c7-9b29-a3cf6eff982f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:37.176196 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08aed8ad-6c06-44c7-9b29-a3cf6eff982f admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:37.176555 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08aed8ad-6c06-44c7-9b29-a3cf6eff982f admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:37.176837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08aed8ad-6c06-44c7-9b29-a3cf6eff982f admin admin] PATH_INFO: `/v3/projects/b042d1409b9644aea25bfd4023371265` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:37.177982 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08aed8ad-6c06-44c7-9b29-a3cf6eff982f admin admin] RBAC: Authorizing `identity:delete_project(project_id=b042d1409b9644aea25bfd4023371265)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:37.179945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08aed8ad-6c06-44c7-9b29-a3cf6eff982f admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:37.233308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-08aed8ad-6c06-44c7-9b29-a3cf6eff982f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b042d1409b9644aea25bfd4023371265', 'request_id': 'req-08aed8ad-6c06-44c7-9b29-a3cf6eff982f'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:50:37.240132 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-08aed8ad-6c06-44c7-9b29-a3cf6eff982f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b042d1409b9644aea25bfd4023371265', 'request_id': 'req-08aed8ad-6c06-44c7-9b29-a3cf6eff982f'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:50:37.245945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-08aed8ad-6c06-44c7-9b29-a3cf6eff982f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b042d1409b9644aea25bfd4023371265', 'request_id': 'req-08aed8ad-6c06-44c7-9b29-a3cf6eff982f'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:50:37.260819 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-08aed8ad-6c06-44c7-9b29-a3cf6eff982f admin admin] The token cache is being invalidate because project b042d1409b9644aea25bfd4023371265 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 15:50:37.261118 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-08aed8ad-6c06-44c7-9b29-a3cf6eff982f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:50:37.261784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-08aed8ad-6c06-44c7-9b29-a3cf6eff982f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 15:50:37.263370 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1129/2233] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 15:50:37 2022] DELETE /identity/v3/projects/b042d1409b9644aea25bfd4023371265 => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:50:37.268894 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56819c82-26c3-4a58-a8c7-c61a5ad76c7e None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:37.269198 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56819c82-26c3-4a58-a8c7-c61a5ad76c7e None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:37.269459 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56819c82-26c3-4a58-a8c7-c61a5ad76c7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:37.292776 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-56819c82-26c3-4a58-a8c7-c61a5ad76c7e None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:50:37.294640 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-56819c82-26c3-4a58-a8c7-c61a5ad76c7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:50:37.357727 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1105/2234] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:50:37 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:50:37.367427 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9eaf036-f2f2-4290-8ff3-f87e4b4aa8b3 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:37.368011 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9eaf036-f2f2-4290-8ff3-f87e4b4aa8b3 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:37.368308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9eaf036-f2f2-4290-8ff3-f87e4b4aa8b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:37.390129 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-b9eaf036-f2f2-4290-8ff3-f87e4b4aa8b3 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:50:37.391533 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-b9eaf036-f2f2-4290-8ff3-f87e4b4aa8b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:50:37.400155 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1130/2235] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:50:37 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:50:37.405191 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56819c82-26c3-4a58-a8c7-c61a5ad76c7e None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:37.414906 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56819c82-26c3-4a58-a8c7-c61a5ad76c7e None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:37.420022 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc866121-465d-42cf-b157-652427a7556e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:37.420350 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc866121-465d-42cf-b157-652427a7556e admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:37.420631 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc866121-465d-42cf-b157-652427a7556e admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:37.420901 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc866121-465d-42cf-b157-652427a7556e admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:37.421557 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc866121-465d-42cf-b157-652427a7556e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 15:50:37.421863 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc866121-465d-42cf-b157-652427a7556e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:37.423497 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc866121-465d-42cf-b157-652427a7556e admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:37.436915 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1106/2236] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 15:50:37 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 15:50:37.441547 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9eaf036-f2f2-4290-8ff3-f87e4b4aa8b3 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:37.452574 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9eaf036-f2f2-4290-8ff3-f87e4b4aa8b3 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:37.458098 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526c7e01-9666-47c6-9d30-eec6a7c1cb3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:37.458717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526c7e01-9666-47c6-9d30-eec6a7c1cb3b admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:37.459002 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526c7e01-9666-47c6-9d30-eec6a7c1cb3b admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:37.459276 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526c7e01-9666-47c6-9d30-eec6a7c1cb3b admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:37.460228 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-526c7e01-9666-47c6-9d30-eec6a7c1cb3b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:37.462885 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-526c7e01-9666-47c6-9d30-eec6a7c1cb3b admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:37.499911 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1131/2237] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:50:37 2022] POST /identity/v3/projects => generated 386 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:50:37.501437 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc866121-465d-42cf-b157-652427a7556e admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:37.514553 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc866121-465d-42cf-b157-652427a7556e admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:37.607884 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5328aaca-1ba7-4c47-b7e5-b77d34253cf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:37.608583 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5328aaca-1ba7-4c47-b7e5-b77d34253cf8 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:37.608964 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5328aaca-1ba7-4c47-b7e5-b77d34253cf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:37.609343 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5328aaca-1ba7-4c47-b7e5-b77d34253cf8 admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:37.610088 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5328aaca-1ba7-4c47-b7e5-b77d34253cf8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:37.612286 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5328aaca-1ba7-4c47-b7e5-b77d34253cf8 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:37.645110 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1107/2238] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:50:37 2022] POST /identity/v3/users => generated 294 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:50:37.646418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526c7e01-9666-47c6-9d30-eec6a7c1cb3b admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:37.659441 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526c7e01-9666-47c6-9d30-eec6a7c1cb3b admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:37.664852 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2d8dc8-5a73-47a5-91b7-223a1f8ab6d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:37.665627 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2d8dc8-5a73-47a5-91b7-223a1f8ab6d4 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:37.665916 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2d8dc8-5a73-47a5-91b7-223a1f8ab6d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:37.666185 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2d8dc8-5a73-47a5-91b7-223a1f8ab6d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:37.667314 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae2d8dc8-5a73-47a5-91b7-223a1f8ab6d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:37.669036 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae2d8dc8-5a73-47a5-91b7-223a1f8ab6d4 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:37.689264 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1132/2239] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:50:37 2022] GET /identity/v3/roles => generated 1427 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:50:37.690610 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5328aaca-1ba7-4c47-b7e5-b77d34253cf8 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:37.702837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5328aaca-1ba7-4c47-b7e5-b77d34253cf8 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:37.709068 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee49d58-d0ce-4e1b-8ba2-ceea61483628 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:37.709735 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee49d58-d0ce-4e1b-8ba2-ceea61483628 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:37.710061 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee49d58-d0ce-4e1b-8ba2-ceea61483628 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:37.710383 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee49d58-d0ce-4e1b-8ba2-ceea61483628 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:37.711041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee49d58-d0ce-4e1b-8ba2-ceea61483628 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:37.713324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee49d58-d0ce-4e1b-8ba2-ceea61483628 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:37.726768 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1108/2240] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:50:37 2022] GET /identity/v3/roles => generated 1427 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 15:50:37.737006 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2d8dc8-5a73-47a5-91b7-223a1f8ab6d4 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:37.751812 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2d8dc8-5a73-47a5-91b7-223a1f8ab6d4 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:37.756531 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c62e7e-252e-4f40-a1f9-0eac6fbfc426 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:37.757246 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c62e7e-252e-4f40-a1f9-0eac6fbfc426 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:37.757482 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c62e7e-252e-4f40-a1f9-0eac6fbfc426 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:37.757755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c62e7e-252e-4f40-a1f9-0eac6fbfc426 admin admin] PATH_INFO: `/v3/projects/e5d926b1275d40fc9b26912ea4bbca02/users/fb26df0d1b82462bb494bd96dba4cf63/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:37.774744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c62e7e-252e-4f40-a1f9-0eac6fbfc426 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e5d926b1275d40fc9b26912ea4bbca02, user_id=fb26df0d1b82462bb494bd96dba4cf63, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:37.777818 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c62e7e-252e-4f40-a1f9-0eac6fbfc426 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:37.791927 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1133/2241] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:50:37 2022] PUT /identity/v3/projects/e5d926b1275d40fc9b26912ea4bbca02/users/fb26df0d1b82462bb494bd96dba4cf63/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:50:37.797270 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509c195e-4c4b-4455-b97e-5116b88a8128 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:37.797558 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509c195e-4c4b-4455-b97e-5116b88a8128 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:37.797833 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509c195e-4c4b-4455-b97e-5116b88a8128 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:37.818139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-509c195e-4c4b-4455-b97e-5116b88a8128 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:50:37.819361 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-509c195e-4c4b-4455-b97e-5116b88a8128 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:50:37.851373 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:50:37.891295 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1109/2242] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:50:37 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:50:37.901774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c62e7e-252e-4f40-a1f9-0eac6fbfc426 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:37.905040 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-d3c62e7e-252e-4f40-a1f9-0eac6fbfc426 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:50:37.980805 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c62e7e-252e-4f40-a1f9-0eac6fbfc426 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:37.986583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4a7953-bcfb-4e93-b8df-509f2d06873e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:37.987463 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4a7953-bcfb-4e93-b8df-509f2d06873e service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:37.987749 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4a7953-bcfb-4e93-b8df-509f2d06873e service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:37.988032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4a7953-bcfb-4e93-b8df-509f2d06873e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:38.000427 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4a7953-bcfb-4e93-b8df-509f2d06873e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:38.002126 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4a7953-bcfb-4e93-b8df-509f2d06873e service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:38.015803 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1134/2243] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:50:37 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:50:38.852950 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409d637c-c311-48be-8ee7-0288e484da1a None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:38.853391 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409d637c-c311-48be-8ee7-0288e484da1a None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:38.853502 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409d637c-c311-48be-8ee7-0288e484da1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:38.878036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-409d637c-c311-48be-8ee7-0288e484da1a None None] MFA Rules not processed for user `ee49fabdcd304c72afefa48a25644bae`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:50:38.879266 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-409d637c-c311-48be-8ee7-0288e484da1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:50:38.971252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1110/2244] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:50:38 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:50:43.500551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd58b0cf-3a27-4bb5-b413-a42ac0e9c6d1 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:43.501026 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd58b0cf-3a27-4bb5-b413-a42ac0e9c6d1 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:43.501120 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd58b0cf-3a27-4bb5-b413-a42ac0e9c6d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:43.565242 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-cd58b0cf-3a27-4bb5-b413-a42ac0e9c6d1 None None] MFA Rules not processed for user `fb26df0d1b82462bb494bd96dba4cf63`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:50:43.566682 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-cd58b0cf-3a27-4bb5-b413-a42ac0e9c6d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:50:43.578738 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:50:43.681125 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1135/2245] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:50:43 2022] POST /identity/v3/auth/tokens => generated 3272 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:50:43.685740 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409d637c-c311-48be-8ee7-0288e484da1a None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:43.702519 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409d637c-c311-48be-8ee7-0288e484da1a None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:43.708164 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ee99f0-3b7f-44f2-85f5-3221a93caf41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:43.708872 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ee99f0-3b7f-44f2-85f5-3221a93caf41 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:43.709246 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ee99f0-3b7f-44f2-85f5-3221a93caf41 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:43.709590 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ee99f0-3b7f-44f2-85f5-3221a93caf41 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:43.710324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ee99f0-3b7f-44f2-85f5-3221a93caf41 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:43.712554 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ee99f0-3b7f-44f2-85f5-3221a93caf41 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:43.745236 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1111/2246] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 15:50:43 2022] POST /identity/v3/projects => generated 384 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:50:43.750595 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd58b0cf-3a27-4bb5-b413-a42ac0e9c6d1 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:43.762488 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd58b0cf-3a27-4bb5-b413-a42ac0e9c6d1 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:43.844003 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e64bb7b-f6c6-4735-a445-d4d5ff6bf6df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:43.844403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e64bb7b-f6c6-4735-a445-d4d5ff6bf6df admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:43.844715 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e64bb7b-f6c6-4735-a445-d4d5ff6bf6df admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:43.844951 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e64bb7b-f6c6-4735-a445-d4d5ff6bf6df admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:43.845591 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e64bb7b-f6c6-4735-a445-d4d5ff6bf6df admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:43.849925 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e64bb7b-f6c6-4735-a445-d4d5ff6bf6df admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:43.879476 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1136/2247] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 15:50:43 2022] POST /identity/v3/users => generated 299 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 15:50:43.883606 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ee99f0-3b7f-44f2-85f5-3221a93caf41 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:43.896284 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ee99f0-3b7f-44f2-85f5-3221a93caf41 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:43.901820 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d025d4-6abe-480e-833b-2e01f18feaa1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:43.902519 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d025d4-6abe-480e-833b-2e01f18feaa1 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:43.902804 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d025d4-6abe-480e-833b-2e01f18feaa1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:43.903264 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d025d4-6abe-480e-833b-2e01f18feaa1 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:43.904088 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23d025d4-6abe-480e-833b-2e01f18feaa1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:43.905756 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23d025d4-6abe-480e-833b-2e01f18feaa1 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:43.918326 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1112/2248] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:50:43 2022] GET /identity/v3/roles => generated 1427 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:50:43.924317 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e64bb7b-f6c6-4735-a445-d4d5ff6bf6df admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:43.949010 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e64bb7b-f6c6-4735-a445-d4d5ff6bf6df admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:43.954870 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf7b835-239a-41fd-b4cf-2cb83b151038 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:43.955562 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf7b835-239a-41fd-b4cf-2cb83b151038 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:43.955878 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf7b835-239a-41fd-b4cf-2cb83b151038 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:43.956158 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf7b835-239a-41fd-b4cf-2cb83b151038 admin admin] PATH_INFO: `/v3/projects/7347dae00eb84f3080d0161cc5112e20/users/bcafd0aed38b4b718ee38614f1c8c9e6/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:43.973466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf7b835-239a-41fd-b4cf-2cb83b151038 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7347dae00eb84f3080d0161cc5112e20, user_id=bcafd0aed38b4b718ee38614f1c8c9e6, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:43.976349 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf7b835-239a-41fd-b4cf-2cb83b151038 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:43.992310 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1137/2249] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:50:43 2022] PUT /identity/v3/projects/7347dae00eb84f3080d0161cc5112e20/users/bcafd0aed38b4b718ee38614f1c8c9e6/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:50:43.996406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d025d4-6abe-480e-833b-2e01f18feaa1 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:44.014905 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d025d4-6abe-480e-833b-2e01f18feaa1 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:44.125980 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af220d2d-87a5-4349-abb6-b0021fb787cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:44.141844 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af220d2d-87a5-4349-abb6-b0021fb787cf admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:44.141844 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af220d2d-87a5-4349-abb6-b0021fb787cf admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:44.141844 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af220d2d-87a5-4349-abb6-b0021fb787cf admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:44.143954 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af220d2d-87a5-4349-abb6-b0021fb787cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:44.147831 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af220d2d-87a5-4349-abb6-b0021fb787cf admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:44.152154 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f854400c-2a0a-4037-81ca-0f9d50947919 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:44.152444 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f854400c-2a0a-4037-81ca-0f9d50947919 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:44.153856 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f854400c-2a0a-4037-81ca-0f9d50947919 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:44.175148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1113/2250] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 15:50:43 2022] GET /identity/v3/roles => generated 1427 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 15:50:44.182678 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af220d2d-87a5-4349-abb6-b0021fb787cf admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:44.202523 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af220d2d-87a5-4349-abb6-b0021fb787cf admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:44.204793 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-f854400c-2a0a-4037-81ca-0f9d50947919 None None] MFA Rules not processed for user `a6c17620ead14cb49d962990e1b1e0d8`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:50:44.206017 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-f854400c-2a0a-4037-81ca-0f9d50947919 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:50:44.208091 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652787eb-c05e-401a-b702-00ffed3eaf9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:44.208844 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652787eb-c05e-401a-b702-00ffed3eaf9b admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:44.209125 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652787eb-c05e-401a-b702-00ffed3eaf9b admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:44.209397 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652787eb-c05e-401a-b702-00ffed3eaf9b admin admin] PATH_INFO: `/v3/projects/7347dae00eb84f3080d0161cc5112e20/users/bcafd0aed38b4b718ee38614f1c8c9e6/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:44.211358 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652787eb-c05e-401a-b702-00ffed3eaf9b admin admin] RBAC: Authorizing `identity:create_grant(project_id=7347dae00eb84f3080d0161cc5112e20, user_id=bcafd0aed38b4b718ee38614f1c8c9e6, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:44.215928 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652787eb-c05e-401a-b702-00ffed3eaf9b admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:44.255960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1114/2251] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 15:50:44 2022] PUT /identity/v3/projects/7347dae00eb84f3080d0161cc5112e20/users/bcafd0aed38b4b718ee38614f1c8c9e6/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 15:50:44.314982 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1138/2252] 10.222.0.102 () {58 vars in 972 bytes} [Wed Aug 24 15:50:44 2022] POST /identity/v3/auth/tokens => generated 3205 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:50:44.344360 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652787eb-c05e-401a-b702-00ffed3eaf9b admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:44.349245 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-652787eb-c05e-401a-b702-00ffed3eaf9b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:50:44.388055 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:50:44.490582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652787eb-c05e-401a-b702-00ffed3eaf9b admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:44.504531 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847b967a-578e-498c-96ce-05e261854749 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a070d4fd8c4466c857bd03965c11349', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:44.505580 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847b967a-578e-498c-96ce-05e261854749 service swift] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:44.505922 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847b967a-578e-498c-96ce-05e261854749 service swift] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:44.506762 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847b967a-578e-498c-96ce-05e261854749 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:44.578858 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847b967a-578e-498c-96ce-05e261854749 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:44.581793 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847b967a-578e-498c-96ce-05e261854749 service swift] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:44.594601 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1115/2253] 10.222.0.102 () {60 vars in 1389 bytes} [Wed Aug 24 15:50:44 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 250 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 24 15:50:49.791066 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556b74c5-b934-45ce-8b45-79eb2c912f62 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:49.791535 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556b74c5-b934-45ce-8b45-79eb2c912f62 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:49.792739 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556b74c5-b934-45ce-8b45-79eb2c912f62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:49.835395 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-556b74c5-b934-45ce-8b45-79eb2c912f62 None None] MFA Rules not processed for user `bcafd0aed38b4b718ee38614f1c8c9e6`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:50:49.836739 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-556b74c5-b934-45ce-8b45-79eb2c912f62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:50:49.850129 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:50:49.924264 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1139/2254] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:50:49 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:50:49.937847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847b967a-578e-498c-96ce-05e261854749 service swift] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:49.939393 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-847b967a-578e-498c-96ce-05e261854749 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:50:50.005580 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847b967a-578e-498c-96ce-05e261854749 service swift] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:50.009399 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b3e20da-a192-4336-bac4-98dc4677a288 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c6f06186c1a4626997db5a40c31cbb9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:50.009917 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3e20da-a192-4336-bac4-98dc4677a288 service glance] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:50.010205 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3e20da-a192-4336-bac4-98dc4677a288 service glance] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:50.010476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3e20da-a192-4336-bac4-98dc4677a288 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:50.072585 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b3e20da-a192-4336-bac4-98dc4677a288 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:50.073952 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b3e20da-a192-4336-bac4-98dc4677a288 service glance] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:50.084351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1116/2255] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:50:49 2022] GET /identity/v3/auth/tokens => generated 3272 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 15:50:50.240234 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5cbb52-3588-4135-ab39-0cdab26c9470 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:50.240769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5cbb52-3588-4135-ab39-0cdab26c9470 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:50.241202 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5cbb52-3588-4135-ab39-0cdab26c9470 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:50.264841 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-ce5cbb52-3588-4135-ab39-0cdab26c9470 None None] MFA Rules not processed for user `a6c17620ead14cb49d962990e1b1e0d8`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:50:50.266389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-ce5cbb52-3588-4135-ab39-0cdab26c9470 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:50:50.274378 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1140/2256] 10.222.0.102 () {58 vars in 1014 bytes} [Wed Aug 24 15:50:50 2022] POST /identity/v3/auth/tokens => generated 3205 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:50:50.289731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b3e20da-a192-4336-bac4-98dc4677a288 service glance] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:50:50.304755 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b3e20da-a192-4336-bac4-98dc4677a288 service glance] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:50:50.310860 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf374dfb-4b34-4ff8-91c5-201daf9dee6e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a070d4fd8c4466c857bd03965c11349', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:50:50.311761 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf374dfb-4b34-4ff8-91c5-201daf9dee6e service swift] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:50.312379 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf374dfb-4b34-4ff8-91c5-201daf9dee6e service swift] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:50.312757 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf374dfb-4b34-4ff8-91c5-201daf9dee6e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:50.322768 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf374dfb-4b34-4ff8-91c5-201daf9dee6e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:50:50.324680 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf374dfb-4b34-4ff8-91c5-201daf9dee6e service swift] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:50:50.335396 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1117/2257] 10.222.0.102 () {60 vars in 1389 bytes} [Wed Aug 24 15:50:50 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 24 15:50:56.728734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b99c70-f417-46eb-8507-a540c1fecb53 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:50:56.729186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b99c70-f417-46eb-8507-a540c1fecb53 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:50:56.729186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b99c70-f417-46eb-8507-a540c1fecb53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:50:56.771890 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-71b99c70-f417-46eb-8507-a540c1fecb53 None None] MFA Rules not processed for user `127f047b72c443eab061a22b1155e053`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:50:56.773117 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-71b99c70-f417-46eb-8507-a540c1fecb53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:50:56.871319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1141/2258] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:50:56 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 15:51:00.107940 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b111dc10-79f1-4e3b-a65a-2b85df2ef5d1 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:51:00.108644 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b111dc10-79f1-4e3b-a65a-2b85df2ef5d1 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:51:00.108751 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b111dc10-79f1-4e3b-a65a-2b85df2ef5d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:51:00.116778 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:51:00.133689 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-b111dc10-79f1-4e3b-a65a-2b85df2ef5d1 None None] MFA Rules not processed for user `fb26df0d1b82462bb494bd96dba4cf63`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:51:00.134940 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-b111dc10-79f1-4e3b-a65a-2b85df2ef5d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:51:00.142554 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1118/2259] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:51:00 2022] POST /identity/v3/auth/tokens => generated 3272 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:51:02.461424 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bdff41-e798-43c1-9b8d-8dc83f086756 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:51:02.461968 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bdff41-e798-43c1-9b8d-8dc83f086756 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:51:02.461968 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bdff41-e798-43c1-9b8d-8dc83f086756 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:51:02.483204 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-15bdff41-e798-43c1-9b8d-8dc83f086756 None None] MFA Rules not processed for user `a6c17620ead14cb49d962990e1b1e0d8`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:51:02.484548 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-15bdff41-e798-43c1-9b8d-8dc83f086756 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:51:02.492520 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1142/2260] 10.222.0.102 () {58 vars in 1014 bytes} [Wed Aug 24 15:51:02 2022] POST /identity/v3/auth/tokens => generated 3205 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:51:02.506501 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b111dc10-79f1-4e3b-a65a-2b85df2ef5d1 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:51:02.520344 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b111dc10-79f1-4e3b-a65a-2b85df2ef5d1 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:51:02.525518 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9eea61-747d-44d4-99e7-3a28578bc34b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a070d4fd8c4466c857bd03965c11349', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:51:02.526133 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9eea61-747d-44d4-99e7-3a28578bc34b service swift] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:51:02.526417 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9eea61-747d-44d4-99e7-3a28578bc34b service swift] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:51:02.526694 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9eea61-747d-44d4-99e7-3a28578bc34b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:51:02.536905 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9eea61-747d-44d4-99e7-3a28578bc34b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:51:02.538565 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9eea61-747d-44d4-99e7-3a28578bc34b service swift] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:51:02.548640 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1119/2261] 10.222.0.102 () {60 vars in 1389 bytes} [Wed Aug 24 15:51:02 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 24 15:51:12.476974 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=55 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:51:12.545031 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=8 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:51:48.998132 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15bdff41-e798-43c1-9b8d-8dc83f086756 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:51:49.053559 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15bdff41-e798-43c1-9b8d-8dc83f086756 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:51:49.057255 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ed8c3e-1c96-4c43-8a7d-e717405b1cf8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:51:49.057855 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ed8c3e-1c96-4c43-8a7d-e717405b1cf8 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:51:49.058135 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ed8c3e-1c96-4c43-8a7d-e717405b1cf8 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:51:49.058408 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ed8c3e-1c96-4c43-8a7d-e717405b1cf8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:51:49.060179 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-12ed8c3e-1c96-4c43-8a7d-e717405b1cf8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:51:49.133760 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ed8c3e-1c96-4c43-8a7d-e717405b1cf8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:51:49.135238 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ed8c3e-1c96-4c43-8a7d-e717405b1cf8 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:51:49.148567 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1143/2262] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:51:48 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:51:57.433990 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9eea61-747d-44d4-99e7-3a28578bc34b service swift] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:51:57.443417 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9eea61-747d-44d4-99e7-3a28578bc34b service swift] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:51:57.447034 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78142d12-14e2-4dcc-b47b-7842d8269cdf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:51:57.447620 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78142d12-14e2-4dcc-b47b-7842d8269cdf service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:51:57.447931 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78142d12-14e2-4dcc-b47b-7842d8269cdf service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:51:57.448203 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78142d12-14e2-4dcc-b47b-7842d8269cdf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:51:57.450088 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-78142d12-14e2-4dcc-b47b-7842d8269cdf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:51:57.510034 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78142d12-14e2-4dcc-b47b-7842d8269cdf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:51:57.511557 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78142d12-14e2-4dcc-b47b-7842d8269cdf service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:51:57.520930 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1120/2263] 10.222.0.102 () {60 vars in 1401 bytes} [Wed Aug 24 15:51:57 2022] GET /identity/v3/auth/tokens?allow_expired=1 => generated 3278 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:51:59.141622 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=54 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:52:03.766344 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ed8c3e-1c96-4c43-8a7d-e717405b1cf8 service neutron] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:52:03.775568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ed8c3e-1c96-4c43-8a7d-e717405b1cf8 service neutron] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:52:03.779262 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389d7cf2-7ab5-4a99-b45a-e44f071e99b8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:52:03.779885 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389d7cf2-7ab5-4a99-b45a-e44f071e99b8 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:52:03.780171 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389d7cf2-7ab5-4a99-b45a-e44f071e99b8 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:52:03.780445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389d7cf2-7ab5-4a99-b45a-e44f071e99b8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:52:03.782165 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-389d7cf2-7ab5-4a99-b45a-e44f071e99b8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:52:03.796129 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-389d7cf2-7ab5-4a99-b45a-e44f071e99b8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:52:03.797555 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-389d7cf2-7ab5-4a99-b45a-e44f071e99b8 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:52:03.808509 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1144/2264] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:52:03 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:52:04.053320 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78142d12-14e2-4dcc-b47b-7842d8269cdf service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:52:04.054857 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-78142d12-14e2-4dcc-b47b-7842d8269cdf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:52:04.071563 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78142d12-14e2-4dcc-b47b-7842d8269cdf service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:52:04.076990 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0119ec4f-515c-4f06-9ad8-13b8ccadae11 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:52:04.077583 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0119ec4f-515c-4f06-9ad8-13b8ccadae11 service nova] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:52:04.077869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0119ec4f-515c-4f06-9ad8-13b8ccadae11 service nova] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:52:04.078136 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0119ec4f-515c-4f06-9ad8-13b8ccadae11 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:52:04.080081 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-0119ec4f-515c-4f06-9ad8-13b8ccadae11 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:52:04.139054 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0119ec4f-515c-4f06-9ad8-13b8ccadae11 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:52:04.140667 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0119ec4f-515c-4f06-9ad8-13b8ccadae11 service nova] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:52:04.150201 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1121/2265] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:52:04 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:52:13.803769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=6 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:52:14.146052 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=56 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:52:30.855809 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389d7cf2-7ab5-4a99-b45a-e44f071e99b8 service neutron] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:52:30.857243 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-389d7cf2-7ab5-4a99-b45a-e44f071e99b8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:52:30.919286 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389d7cf2-7ab5-4a99-b45a-e44f071e99b8 service neutron] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:52:30.922970 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77895b52-0f0b-4a70-b235-66ae1b9cecc1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:52:30.923616 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77895b52-0f0b-4a70-b235-66ae1b9cecc1 service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:52:30.923927 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77895b52-0f0b-4a70-b235-66ae1b9cecc1 service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:52:30.924207 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77895b52-0f0b-4a70-b235-66ae1b9cecc1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:52:30.926039 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-77895b52-0f0b-4a70-b235-66ae1b9cecc1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:52:30.986885 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77895b52-0f0b-4a70-b235-66ae1b9cecc1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:52:30.988408 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77895b52-0f0b-4a70-b235-66ae1b9cecc1 service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:52:30.997682 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1145/2266] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:52:30 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:52:39.142399 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0119ec4f-515c-4f06-9ad8-13b8ccadae11 service nova] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:52:39.143986 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-0119ec4f-515c-4f06-9ad8-13b8ccadae11 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:52:39.156206 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0119ec4f-515c-4f06-9ad8-13b8ccadae11 service nova] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:52:39.160832 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df03e19-8ccc-4dce-8bb9-e8b5e9cf8236 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:52:39.161879 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df03e19-8ccc-4dce-8bb9-e8b5e9cf8236 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:52:39.162369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df03e19-8ccc-4dce-8bb9-e8b5e9cf8236 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:52:39.162964 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df03e19-8ccc-4dce-8bb9-e8b5e9cf8236 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:52:39.165818 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-8df03e19-8ccc-4dce-8bb9-e8b5e9cf8236 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:52:39.177050 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df03e19-8ccc-4dce-8bb9-e8b5e9cf8236 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:52:39.178656 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df03e19-8ccc-4dce-8bb9-e8b5e9cf8236 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:52:39.196301 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1122/2267] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:52:39 2022] GET /identity/v3/auth/tokens => generated 3131 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:52:40.993637 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=48 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:52:41.093138 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77895b52-0f0b-4a70-b235-66ae1b9cecc1 service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:52:41.104005 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77895b52-0f0b-4a70-b235-66ae1b9cecc1 service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:52:41.108063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca22e0f-070d-456d-8192-5d4e6e9bef5d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:52:41.108876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca22e0f-070d-456d-8192-5d4e6e9bef5d service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:52:41.109347 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca22e0f-070d-456d-8192-5d4e6e9bef5d service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:52:41.109777 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca22e0f-070d-456d-8192-5d4e6e9bef5d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:52:41.121395 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca22e0f-070d-456d-8192-5d4e6e9bef5d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:52:41.123108 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca22e0f-070d-456d-8192-5d4e6e9bef5d service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:52:41.135064 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1146/2268] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:52:41 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:52:42.204365 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df03e19-8ccc-4dce-8bb9-e8b5e9cf8236 service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:52:42.230253 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df03e19-8ccc-4dce-8bb9-e8b5e9cf8236 service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:52:42.244310 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07644bf1-32c5-41f1-a112-966009d253d5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:52:42.245959 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07644bf1-32c5-41f1-a112-966009d253d5 service nova] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:52:42.246248 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07644bf1-32c5-41f1-a112-966009d253d5 service nova] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:52:42.246519 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07644bf1-32c5-41f1-a112-966009d253d5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:52:42.252361 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-07644bf1-32c5-41f1-a112-966009d253d5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:52:42.284163 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07644bf1-32c5-41f1-a112-966009d253d5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:52:42.287938 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07644bf1-32c5-41f1-a112-966009d253d5 service nova] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:52:42.322317 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1123/2269] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:52:42 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:52:44.251801 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca22e0f-070d-456d-8192-5d4e6e9bef5d service neutron] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:52:44.253944 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-3ca22e0f-070d-456d-8192-5d4e6e9bef5d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:52:44.286260 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca22e0f-070d-456d-8192-5d4e6e9bef5d service neutron] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:52:44.305876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fcb5a0-0dac-4ef2-a263-a352dce9efb5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:52:44.314459 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fcb5a0-0dac-4ef2-a263-a352dce9efb5 service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:52:44.314918 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fcb5a0-0dac-4ef2-a263-a352dce9efb5 service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:52:44.315297 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fcb5a0-0dac-4ef2-a263-a352dce9efb5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:52:44.324628 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-12fcb5a0-0dac-4ef2-a263-a352dce9efb5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:52:44.356547 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12fcb5a0-0dac-4ef2-a263-a352dce9efb5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:52:44.359302 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12fcb5a0-0dac-4ef2-a263-a352dce9efb5 service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:52:44.377122 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1147/2270] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:52:44 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:52:47.419186 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb9bacb-ab5a-4778-bb00-414a9976829b None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:52:47.419800 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb9bacb-ab5a-4778-bb00-414a9976829b None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:52:47.419800 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb9bacb-ab5a-4778-bb00-414a9976829b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:52:47.445968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-dfb9bacb-ab5a-4778-bb00-414a9976829b None None] MFA Rules not processed for user `2fe5724620da46ddbefbdfd442dc8d73`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:52:47.453321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-dfb9bacb-ab5a-4778-bb00-414a9976829b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:52:47.464525 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1124/2271] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:52:47 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:52:48.244501 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610220af-83e7-462f-8146-94fc0bfba6de None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:52:48.245185 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610220af-83e7-462f-8146-94fc0bfba6de None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:52:48.245185 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610220af-83e7-462f-8146-94fc0bfba6de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:52:48.269393 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-610220af-83e7-462f-8146-94fc0bfba6de None None] MFA Rules not processed for user `38f7d31cf0f541f58af7d3d850af6472`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 15:52:48.270697 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-610220af-83e7-462f-8146-94fc0bfba6de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:52:48.278818 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1148/2272] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 15:52:48 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 15:52:48.612663 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb9bacb-ab5a-4778-bb00-414a9976829b None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:52:48.625111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb9bacb-ab5a-4778-bb00-414a9976829b None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:52:48.628829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6fa82f-1e29-49bb-8657-a85a6721ed3b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:52:48.629438 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6fa82f-1e29-49bb-8657-a85a6721ed3b service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:52:48.629741 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6fa82f-1e29-49bb-8657-a85a6721ed3b service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:52:48.632867 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6fa82f-1e29-49bb-8657-a85a6721ed3b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:52:48.635846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-bc6fa82f-1e29-49bb-8657-a85a6721ed3b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:52:48.646044 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6fa82f-1e29-49bb-8657-a85a6721ed3b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:52:48.647482 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6fa82f-1e29-49bb-8657-a85a6721ed3b service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:52:48.658570 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1125/2273] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:52:48 2022] GET /identity/v3/auth/tokens => generated 3131 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:52:49.000325 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610220af-83e7-462f-8146-94fc0bfba6de None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:52:49.001803 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-610220af-83e7-462f-8146-94fc0bfba6de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:52:49.107913 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610220af-83e7-462f-8146-94fc0bfba6de None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:52:49.113995 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5cf92f-2efe-4a30-a7ce-6de7ab17afa3 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0ee0002a76c3471f819b37fef510ef99', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:52:49.114737 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5cf92f-2efe-4a30-a7ce-6de7ab17afa3 service placement] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:52:49.115134 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5cf92f-2efe-4a30-a7ce-6de7ab17afa3 service placement] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:52:49.115568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5cf92f-2efe-4a30-a7ce-6de7ab17afa3 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:52:49.117617 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-af5cf92f-2efe-4a30-a7ce-6de7ab17afa3 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:52:49.130043 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af5cf92f-2efe-4a30-a7ce-6de7ab17afa3 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:52:49.131657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af5cf92f-2efe-4a30-a7ce-6de7ab17afa3 service placement] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:52:49.144526 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1149/2274] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:52:48 2022] GET /identity/v3/auth/tokens => generated 3318 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:52:58.653993 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=21 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:52:59.139968 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=47 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:53:01.148704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6fa82f-1e29-49bb-8657-a85a6721ed3b service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:53:01.163262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6fa82f-1e29-49bb-8657-a85a6721ed3b service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:53:01.169939 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831d49ed-163b-4399-8220-b435967fad16 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:53:01.170552 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831d49ed-163b-4399-8220-b435967fad16 service nova] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:53:01.170833 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831d49ed-163b-4399-8220-b435967fad16 service nova] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:53:01.171105 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831d49ed-163b-4399-8220-b435967fad16 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:53:01.174725 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-831d49ed-163b-4399-8220-b435967fad16 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:53:01.191019 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831d49ed-163b-4399-8220-b435967fad16 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:53:01.194534 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831d49ed-163b-4399-8220-b435967fad16 service nova] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:53:01.220652 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1126/2275] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:53:01 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 73 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:53:11.202750 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=6 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:53:34.178838 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5cf92f-2efe-4a30-a7ce-6de7ab17afa3 service placement] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:53:34.192015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5cf92f-2efe-4a30-a7ce-6de7ab17afa3 service placement] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:53:34.197361 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58cdae9e-9770-4bae-b2ea-aa8ff63536b1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:53:34.197990 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cdae9e-9770-4bae-b2ea-aa8ff63536b1 service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:53:34.198278 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cdae9e-9770-4bae-b2ea-aa8ff63536b1 service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:53:34.198542 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cdae9e-9770-4bae-b2ea-aa8ff63536b1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:53:34.200258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-58cdae9e-9770-4bae-b2ea-aa8ff63536b1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:53:34.259060 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58cdae9e-9770-4bae-b2ea-aa8ff63536b1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:53:34.260680 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58cdae9e-9770-4bae-b2ea-aa8ff63536b1 service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:53:34.269715 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1150/2276] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:53:34 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:53:39.163333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831d49ed-163b-4399-8220-b435967fad16 service nova] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:53:39.172862 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831d49ed-163b-4399-8220-b435967fad16 service nova] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:53:39.176469 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5928559c-a384-4c08-b4ae-a560a5119d95 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:53:39.177047 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5928559c-a384-4c08-b4ae-a560a5119d95 service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:53:39.177333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5928559c-a384-4c08-b4ae-a560a5119d95 service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:53:39.177606 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5928559c-a384-4c08-b4ae-a560a5119d95 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:53:39.179324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-5928559c-a384-4c08-b4ae-a560a5119d95 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:53:39.190567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5928559c-a384-4c08-b4ae-a560a5119d95 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:53:39.192041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5928559c-a384-4c08-b4ae-a560a5119d95 service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:53:39.201056 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1127/2277] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:53:39 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:53:44.265464 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=28 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:53:49.196867 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=6 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:55:49.937300 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58cdae9e-9770-4bae-b2ea-aa8ff63536b1 service nova] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:55:50.009421 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58cdae9e-9770-4bae-b2ea-aa8ff63536b1 service nova] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:55:50.015233 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73a2cdd-85e6-4b23-8b2a-b1685915b772 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:55:50.016528 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73a2cdd-85e6-4b23-8b2a-b1685915b772 service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:55:50.016817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73a2cdd-85e6-4b23-8b2a-b1685915b772 service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:55:50.017086 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73a2cdd-85e6-4b23-8b2a-b1685915b772 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:55:50.025452 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73a2cdd-85e6-4b23-8b2a-b1685915b772 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:55:50.027039 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73a2cdd-85e6-4b23-8b2a-b1685915b772 service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:55:50.036029 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1151/2278] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:55:49 2022] GET /identity/v3/auth/tokens => generated 3272 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:55:55.385477 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5928559c-a384-4c08-b4ae-a560a5119d95 service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:55:55.387129 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-5928559c-a384-4c08-b4ae-a560a5119d95 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:55:55.468642 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5928559c-a384-4c08-b4ae-a560a5119d95 service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:55:55.475273 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e562ac9-929d-4c59-be1d-5ac97cfd1cf4 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0ee0002a76c3471f819b37fef510ef99', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:55:55.475820 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e562ac9-929d-4c59-be1d-5ac97cfd1cf4 service placement] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:55:55.476099 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e562ac9-929d-4c59-be1d-5ac97cfd1cf4 service placement] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:55:55.476367 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e562ac9-929d-4c59-be1d-5ac97cfd1cf4 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:55:55.478070 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-3e562ac9-929d-4c59-be1d-5ac97cfd1cf4 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 15:55:55.490587 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e562ac9-929d-4c59-be1d-5ac97cfd1cf4 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:55:55.493126 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e562ac9-929d-4c59-be1d-5ac97cfd1cf4 service placement] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:55:55.507923 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1128/2279] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:55:55 2022] GET /identity/v3/auth/tokens => generated 3318 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:56:00.032700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=30 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:56:05.502120 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=30 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:56:58.064672 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73a2cdd-85e6-4b23-8b2a-b1685915b772 service nova] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:56:58.078860 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73a2cdd-85e6-4b23-8b2a-b1685915b772 service nova] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:56:58.085054 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e69ab9-06c0-4b27-8501-dadfe8b3b4df service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:56:58.086128 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e69ab9-06c0-4b27-8501-dadfe8b3b4df service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:56:58.086642 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e69ab9-06c0-4b27-8501-dadfe8b3b4df service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:56:58.086998 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e69ab9-06c0-4b27-8501-dadfe8b3b4df service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:56:58.097596 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e69ab9-06c0-4b27-8501-dadfe8b3b4df service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:56:58.099202 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e69ab9-06c0-4b27-8501-dadfe8b3b4df service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:56:58.129911 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1152/2280] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:56:58 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 66 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:57:03.813485 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e562ac9-929d-4c59-be1d-5ac97cfd1cf4 service placement] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:57:03.826400 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e562ac9-929d-4c59-be1d-5ac97cfd1cf4 service placement] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:57:03.831931 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd46dab-07bc-4c56-90fc-6580893c41bc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:57:03.832531 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd46dab-07bc-4c56-90fc-6580893c41bc service nova] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:57:03.832810 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd46dab-07bc-4c56-90fc-6580893c41bc service nova] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:57:03.833081 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd46dab-07bc-4c56-90fc-6580893c41bc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:57:03.842794 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bd46dab-07bc-4c56-90fc-6580893c41bc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:57:03.844425 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bd46dab-07bc-4c56-90fc-6580893c41bc service nova] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:57:03.873432 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1129/2281] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:57:03 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:57:08.127143 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=13 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:57:13.870427 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=13 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:57:31.293991 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e69ab9-06c0-4b27-8501-dadfe8b3b4df service nova] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:57:31.313201 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e69ab9-06c0-4b27-8501-dadfe8b3b4df service nova] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:57:31.316983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff8ddc3-86ca-4881-b0f2-82ace3ff8e09 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:57:31.317591 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff8ddc3-86ca-4881-b0f2-82ace3ff8e09 service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:57:31.317859 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff8ddc3-86ca-4881-b0f2-82ace3ff8e09 service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:57:31.318123 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff8ddc3-86ca-4881-b0f2-82ace3ff8e09 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:57:31.326950 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff8ddc3-86ca-4881-b0f2-82ace3ff8e09 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:57:31.330298 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff8ddc3-86ca-4881-b0f2-82ace3ff8e09 service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:57:31.357494 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1153/2282] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:57:31 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:57:38.863542 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd46dab-07bc-4c56-90fc-6580893c41bc service nova] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:57:38.873390 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd46dab-07bc-4c56-90fc-6580893c41bc service nova] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:57:38.877024 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ae373c-2d93-4795-bc88-424e4b96b6de service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:57:38.877613 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ae373c-2d93-4795-bc88-424e4b96b6de service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:57:38.877895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ae373c-2d93-4795-bc88-424e4b96b6de service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:57:38.878168 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ae373c-2d93-4795-bc88-424e4b96b6de service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:57:38.889124 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ae373c-2d93-4795-bc88-424e4b96b6de service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:57:38.890568 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ae373c-2d93-4795-bc88-424e4b96b6de service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:57:38.902018 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1130/2283] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:57:38 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:57:41.355430 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=17 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:57:48.896800 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=6 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:57:55.304922 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff8ddc3-86ca-4881-b0f2-82ace3ff8e09 service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:57:55.325714 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff8ddc3-86ca-4881-b0f2-82ace3ff8e09 service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:57:55.329380 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce5b985-4d22-4057-9d97-624a17b95ffd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:57:55.329973 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce5b985-4d22-4057-9d97-624a17b95ffd service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:57:55.330253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce5b985-4d22-4057-9d97-624a17b95ffd service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:57:55.330524 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce5b985-4d22-4057-9d97-624a17b95ffd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:57:55.337719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce5b985-4d22-4057-9d97-624a17b95ffd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:57:55.339168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce5b985-4d22-4057-9d97-624a17b95ffd service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:57:55.347424 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1154/2284] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:57:55 2022] GET /identity/v3/auth/tokens => generated 3131 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:58:05.343811 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=10 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:58:34.004633 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ae373c-2d93-4795-bc88-424e4b96b6de service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:58:34.015871 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ae373c-2d93-4795-bc88-424e4b96b6de service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:58:34.022584 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944a4023-7447-4d58-8194-7cae5a8d0597 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:58:34.023122 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944a4023-7447-4d58-8194-7cae5a8d0597 service nova] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:58:34.023405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944a4023-7447-4d58-8194-7cae5a8d0597 service nova] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:58:34.023708 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944a4023-7447-4d58-8194-7cae5a8d0597 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:58:34.031856 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-944a4023-7447-4d58-8194-7cae5a8d0597 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:58:34.033530 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-944a4023-7447-4d58-8194-7cae5a8d0597 service nova] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:58:34.062769 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1131/2285] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:58:34 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:58:39.391705 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce5b985-4d22-4057-9d97-624a17b95ffd service neutron] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:58:39.402363 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce5b985-4d22-4057-9d97-624a17b95ffd service neutron] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:58:39.405909 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a86f2a-87f3-4897-a366-9ad578e3c814 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:58:39.406504 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a86f2a-87f3-4897-a366-9ad578e3c814 service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:58:39.406780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a86f2a-87f3-4897-a366-9ad578e3c814 service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:58:39.407052 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a86f2a-87f3-4897-a366-9ad578e3c814 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:58:39.416881 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a86f2a-87f3-4897-a366-9ad578e3c814 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:58:39.418797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a86f2a-87f3-4897-a366-9ad578e3c814 service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:58:39.446792 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1155/2286] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:58:39 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 15:58:44.059485 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=13 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:58:49.444079 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=13 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 15:59:51.067969 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944a4023-7447-4d58-8194-7cae5a8d0597 service nova] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 15:59:51.082793 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944a4023-7447-4d58-8194-7cae5a8d0597 service nova] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 15:59:51.088680 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92ed795-6861-4ee4-954f-45c9f5784605 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 15:59:51.089299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92ed795-6861-4ee4-954f-45c9f5784605 service nova] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 15:59:51.089583 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92ed795-6861-4ee4-954f-45c9f5784605 service nova] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 15:59:51.089868 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92ed795-6861-4ee4-954f-45c9f5784605 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 15:59:51.103599 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92ed795-6861-4ee4-954f-45c9f5784605 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 15:59:51.105559 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92ed795-6861-4ee4-954f-45c9f5784605 service nova] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 15:59:51.131744 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1132/2287] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 15:59:51 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:00:01.114220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=6 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:00:49.051939 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a86f2a-87f3-4897-a366-9ad578e3c814 service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:00:49.053351 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-31a86f2a-87f3-4897-a366-9ad578e3c814 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:00:49.077379 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a86f2a-87f3-4897-a366-9ad578e3c814 service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:00:49.081051 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda47cfa-0da6-4e80-b6bc-f45ab976ccf4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:00:49.081650 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda47cfa-0da6-4e80-b6bc-f45ab976ccf4 service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:00:49.081933 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda47cfa-0da6-4e80-b6bc-f45ab976ccf4 service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:00:49.082208 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda47cfa-0da6-4e80-b6bc-f45ab976ccf4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:00:49.084056 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-eda47cfa-0da6-4e80-b6bc-f45ab976ccf4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:00:49.116481 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda47cfa-0da6-4e80-b6bc-f45ab976ccf4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:00:49.117856 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda47cfa-0da6-4e80-b6bc-f45ab976ccf4 service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:00:49.136299 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1156/2288] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:00:49 2022] GET /identity/v3/auth/tokens => generated 3272 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:00:59.133661 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=24 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:00:59.681081 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92ed795-6861-4ee4-954f-45c9f5784605 service nova] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:00:59.693227 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92ed795-6861-4ee4-954f-45c9f5784605 service nova] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:00:59.698152 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606f7864-be5c-4fe8-a24a-d6d2c6a90bad service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0ee0002a76c3471f819b37fef510ef99', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:00:59.698743 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606f7864-be5c-4fe8-a24a-d6d2c6a90bad service placement] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:00:59.699035 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606f7864-be5c-4fe8-a24a-d6d2c6a90bad service placement] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:00:59.699294 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606f7864-be5c-4fe8-a24a-d6d2c6a90bad service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:00:59.710691 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606f7864-be5c-4fe8-a24a-d6d2c6a90bad service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:00:59.714076 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606f7864-be5c-4fe8-a24a-d6d2c6a90bad service placement] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:00:59.728182 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1133/2289] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:00:59 2022] GET /identity/v3/auth/tokens => generated 3318 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:01:09.722223 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=6 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:01:57.251366 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda47cfa-0da6-4e80-b6bc-f45ab976ccf4 service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:01:57.261092 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda47cfa-0da6-4e80-b6bc-f45ab976ccf4 service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:01:57.264935 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5688df00-c9fa-4520-bcfd-1f27625be478 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:01:57.265540 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5688df00-c9fa-4520-bcfd-1f27625be478 service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:01:57.265827 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5688df00-c9fa-4520-bcfd-1f27625be478 service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:01:57.266103 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5688df00-c9fa-4520-bcfd-1f27625be478 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:01:57.275087 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5688df00-c9fa-4520-bcfd-1f27625be478 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:01:57.276731 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5688df00-c9fa-4520-bcfd-1f27625be478 service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:01:57.285642 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1157/2290] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:01:57 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:02:03.683725 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606f7864-be5c-4fe8-a24a-d6d2c6a90bad service placement] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:02:03.693488 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606f7864-be5c-4fe8-a24a-d6d2c6a90bad service placement] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:02:03.698102 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7f69e9-f871-43b2-9dc6-13094c33df33 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:02:03.698722 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7f69e9-f871-43b2-9dc6-13094c33df33 service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:02:03.699005 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7f69e9-f871-43b2-9dc6-13094c33df33 service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:02:03.699283 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7f69e9-f871-43b2-9dc6-13094c33df33 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:02:03.708021 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7f69e9-f871-43b2-9dc6-13094c33df33 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:02:03.709440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7f69e9-f871-43b2-9dc6-13094c33df33 service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:02:03.718527 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1134/2291] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:02:03 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:02:07.281549 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=6 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:02:13.714449 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=6 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:02:30.319948 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5688df00-c9fa-4520-bcfd-1f27625be478 service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:02:30.329606 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5688df00-c9fa-4520-bcfd-1f27625be478 service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:02:30.334021 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9707e51e-19fa-4a8f-9c04-aa4149d48c89 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:02:30.334680 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9707e51e-19fa-4a8f-9c04-aa4149d48c89 service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:02:30.334901 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9707e51e-19fa-4a8f-9c04-aa4149d48c89 service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:02:30.335176 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9707e51e-19fa-4a8f-9c04-aa4149d48c89 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:02:30.343831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9707e51e-19fa-4a8f-9c04-aa4149d48c89 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:02:30.345248 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9707e51e-19fa-4a8f-9c04-aa4149d48c89 service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:02:30.354732 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1158/2292] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:02:30 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 16:02:40.350236 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=6 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:03:05.051448 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7f69e9-f871-43b2-9dc6-13094c33df33 service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:03:05.053056 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-2f7f69e9-f871-43b2-9dc6-13094c33df33 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:03:05.128036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7f69e9-f871-43b2-9dc6-13094c33df33 service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:03:05.131680 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba5e32a-0c7b-4de7-8685-fdddffc29f9a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:03:05.132304 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba5e32a-0c7b-4de7-8685-fdddffc29f9a service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:03:05.132584 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba5e32a-0c7b-4de7-8685-fdddffc29f9a service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:03:05.133004 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba5e32a-0c7b-4de7-8685-fdddffc29f9a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:03:05.134849 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-4ba5e32a-0c7b-4de7-8685-fdddffc29f9a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:03:05.144807 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ba5e32a-0c7b-4de7-8685-fdddffc29f9a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:03:05.146262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ba5e32a-0c7b-4de7-8685-fdddffc29f9a service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:03:05.154256 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1135/2293] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:03:05 2022] GET /identity/v3/auth/tokens => generated 3131 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:03:15.150674 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=31 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:03:33.785726 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9707e51e-19fa-4a8f-9c04-aa4149d48c89 service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:03:33.842761 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9707e51e-19fa-4a8f-9c04-aa4149d48c89 service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:03:33.847478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ef0e22-e0fa-444e-8b21-7f9fc503f8bc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:03:33.847898 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ef0e22-e0fa-444e-8b21-7f9fc503f8bc service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:03:33.848183 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ef0e22-e0fa-444e-8b21-7f9fc503f8bc service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:03:33.848459 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ef0e22-e0fa-444e-8b21-7f9fc503f8bc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:03:33.857192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3ef0e22-e0fa-444e-8b21-7f9fc503f8bc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:03:33.858621 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3ef0e22-e0fa-444e-8b21-7f9fc503f8bc service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:03:33.867704 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1159/2294] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:03:33 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:03:39.208537 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba5e32a-0c7b-4de7-8685-fdddffc29f9a service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:03:39.218125 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba5e32a-0c7b-4de7-8685-fdddffc29f9a service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:03:39.222979 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1404525-a778-4677-8e89-97eb1bd985ab service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:03:39.223560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1404525-a778-4677-8e89-97eb1bd985ab service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:03:39.223862 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1404525-a778-4677-8e89-97eb1bd985ab service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:03:39.224131 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1404525-a778-4677-8e89-97eb1bd985ab service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:03:39.225922 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-d1404525-a778-4677-8e89-97eb1bd985ab service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:03:39.301662 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1404525-a778-4677-8e89-97eb1bd985ab service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:03:39.303106 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1404525-a778-4677-8e89-97eb1bd985ab service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:03:39.312301 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1136/2295] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:03:39 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 16:03:39.568294 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ef0e22-e0fa-444e-8b21-7f9fc503f8bc service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:03:39.569933 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-a3ef0e22-e0fa-444e-8b21-7f9fc503f8bc service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:03:39.581758 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ef0e22-e0fa-444e-8b21-7f9fc503f8bc service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:03:39.585470 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38043b29-1f16-4e3d-8cf7-8c3aabf8e06c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:03:39.586058 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38043b29-1f16-4e3d-8cf7-8c3aabf8e06c service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:03:39.586337 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38043b29-1f16-4e3d-8cf7-8c3aabf8e06c service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:03:39.586607 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38043b29-1f16-4e3d-8cf7-8c3aabf8e06c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:03:39.588406 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-38043b29-1f16-4e3d-8cf7-8c3aabf8e06c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:03:39.674183 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38043b29-1f16-4e3d-8cf7-8c3aabf8e06c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:03:39.675798 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38043b29-1f16-4e3d-8cf7-8c3aabf8e06c service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:03:39.688480 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1160/2296] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:03:39 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:03:40.902174 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1404525-a778-4677-8e89-97eb1bd985ab service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:03:40.912150 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1404525-a778-4677-8e89-97eb1bd985ab service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:03:40.915757 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822226d7-6487-4488-9950-7f1f82e4bd49 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:03:40.916349 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822226d7-6487-4488-9950-7f1f82e4bd49 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:03:40.916629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822226d7-6487-4488-9950-7f1f82e4bd49 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:03:40.916905 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822226d7-6487-4488-9950-7f1f82e4bd49 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:03:40.918881 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-822226d7-6487-4488-9950-7f1f82e4bd49 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:03:40.933259 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-822226d7-6487-4488-9950-7f1f82e4bd49 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:03:40.934698 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-822226d7-6487-4488-9950-7f1f82e4bd49 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:03:40.946715 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1137/2297] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:03:40 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 16:03:49.682679 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=65 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:03:50.941801 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=39 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:05:49.795297 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38043b29-1f16-4e3d-8cf7-8c3aabf8e06c service neutron] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:05:49.805529 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38043b29-1f16-4e3d-8cf7-8c3aabf8e06c service neutron] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:05:49.809122 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a3fc70-179d-4994-b7db-387d1763002b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:05:49.809718 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a3fc70-179d-4994-b7db-387d1763002b service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:05:49.809995 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a3fc70-179d-4994-b7db-387d1763002b service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:05:49.810311 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a3fc70-179d-4994-b7db-387d1763002b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:05:49.867613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a3fc70-179d-4994-b7db-387d1763002b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:05:49.869086 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a3fc70-179d-4994-b7db-387d1763002b service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:05:49.878047 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1161/2298] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:05:49 2022] GET /identity/v3/auth/tokens => generated 3272 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 16:05:59.874027 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=28 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:06:06.656581 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822226d7-6487-4488-9950-7f1f82e4bd49 service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:06:06.658902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-822226d7-6487-4488-9950-7f1f82e4bd49 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:06:06.841885 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822226d7-6487-4488-9950-7f1f82e4bd49 service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:06:06.848843 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1ba435-effe-4310-8d72-95efb4149eaa service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0ee0002a76c3471f819b37fef510ef99', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:06:06.849677 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1ba435-effe-4310-8d72-95efb4149eaa service placement] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:06:06.849802 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1ba435-effe-4310-8d72-95efb4149eaa service placement] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:06:06.849992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1ba435-effe-4310-8d72-95efb4149eaa service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:06:06.851941 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-9d1ba435-effe-4310-8d72-95efb4149eaa service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:06:06.863826 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1ba435-effe-4310-8d72-95efb4149eaa service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:06:06.865645 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1ba435-effe-4310-8d72-95efb4149eaa service placement] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:06:06.904427 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1138/2299] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:06:06 2022] GET /identity/v3/auth/tokens => generated 3318 bytes in 248 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 16:06:16.877619 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=63 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:06:57.720342 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a3fc70-179d-4994-b7db-387d1763002b service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:06:57.722088 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-15a3fc70-179d-4994-b7db-387d1763002b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:06:57.781844 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a3fc70-179d-4994-b7db-387d1763002b service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:06:57.787904 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719247a7-e2b7-4379-8877-26a9057331f3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:06:57.788956 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719247a7-e2b7-4379-8877-26a9057331f3 service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:06:57.788956 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719247a7-e2b7-4379-8877-26a9057331f3 service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:06:57.789195 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719247a7-e2b7-4379-8877-26a9057331f3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:06:57.790993 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-719247a7-e2b7-4379-8877-26a9057331f3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:06:57.874199 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-719247a7-e2b7-4379-8877-26a9057331f3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:06:57.875987 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-719247a7-e2b7-4379-8877-26a9057331f3 service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:06:57.898703 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1162/2300] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:06:57 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:07:03.498159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1ba435-effe-4310-8d72-95efb4149eaa service placement] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:07:03.499811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-9d1ba435-effe-4310-8d72-95efb4149eaa service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:07:03.515157 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1ba435-effe-4310-8d72-95efb4149eaa service placement] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:07:03.520532 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa0a0a8b-3c2c-4390-9c50-5a15016a04f2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:07:03.521828 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0a0a8b-3c2c-4390-9c50-5a15016a04f2 service nova] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:07:03.522108 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0a0a8b-3c2c-4390-9c50-5a15016a04f2 service nova] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:07:03.522379 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0a0a8b-3c2c-4390-9c50-5a15016a04f2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:07:03.524199 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-aa0a0a8b-3c2c-4390-9c50-5a15016a04f2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:07:03.602414 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa0a0a8b-3c2c-4390-9c50-5a15016a04f2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:07:03.604717 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa0a0a8b-3c2c-4390-9c50-5a15016a04f2 service nova] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:07:03.616908 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1139/2301] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:07:03 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:07:07.882560 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=51 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:07:13.611077 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=33 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:07:50.206775 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719247a7-e2b7-4379-8877-26a9057331f3 service nova] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:07:50.227088 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719247a7-e2b7-4379-8877-26a9057331f3 service nova] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:07:50.232041 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fac2fbc-0a1b-488b-90ff-f5261d38ca48 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:07:50.232908 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fac2fbc-0a1b-488b-90ff-f5261d38ca48 service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:07:50.233326 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fac2fbc-0a1b-488b-90ff-f5261d38ca48 service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:07:50.233726 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fac2fbc-0a1b-488b-90ff-f5261d38ca48 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:07:50.236498 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-6fac2fbc-0a1b-488b-90ff-f5261d38ca48 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:07:50.327547 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fac2fbc-0a1b-488b-90ff-f5261d38ca48 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:07:50.329021 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fac2fbc-0a1b-488b-90ff-f5261d38ca48 service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:07:50.348515 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1163/2302] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:07:50 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:08:00.345993 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=45 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:08:34.607778 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa0a0a8b-3c2c-4390-9c50-5a15016a04f2 service nova] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:08:34.623556 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa0a0a8b-3c2c-4390-9c50-5a15016a04f2 service nova] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:08:34.630059 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae80cb06-4da3-4379-9fa2-e61f603e2e0d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:08:34.630689 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae80cb06-4da3-4379-9fa2-e61f603e2e0d service nova] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:08:34.631116 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae80cb06-4da3-4379-9fa2-e61f603e2e0d service nova] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:08:34.631420 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae80cb06-4da3-4379-9fa2-e61f603e2e0d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:08:34.633466 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-ae80cb06-4da3-4379-9fa2-e61f603e2e0d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:08:34.714441 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae80cb06-4da3-4379-9fa2-e61f603e2e0d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:08:34.716131 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae80cb06-4da3-4379-9fa2-e61f603e2e0d service nova] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:08:34.734416 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1140/2303] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:08:34 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:08:38.665588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fac2fbc-0a1b-488b-90ff-f5261d38ca48 service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:08:38.679001 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fac2fbc-0a1b-488b-90ff-f5261d38ca48 service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:08:38.684583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9ad003-a204-437e-bb93-fd7b067df227 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:08:38.685190 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9ad003-a204-437e-bb93-fd7b067df227 service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:08:38.685476 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9ad003-a204-437e-bb93-fd7b067df227 service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:08:38.685753 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9ad003-a204-437e-bb93-fd7b067df227 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:08:38.694590 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a9ad003-a204-437e-bb93-fd7b067df227 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:08:38.696210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a9ad003-a204-437e-bb93-fd7b067df227 service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:08:38.705538 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1164/2304] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:08:38 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:08:39.964607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae80cb06-4da3-4379-9fa2-e61f603e2e0d service nova] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:08:39.984798 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae80cb06-4da3-4379-9fa2-e61f603e2e0d service nova] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:08:39.988619 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6073f9d2-e99f-4f36-a31b-2951697e2bb6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:08:39.990456 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6073f9d2-e99f-4f36-a31b-2951697e2bb6 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:08:39.990806 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6073f9d2-e99f-4f36-a31b-2951697e2bb6 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:08:39.991184 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6073f9d2-e99f-4f36-a31b-2951697e2bb6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:08:40.001987 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6073f9d2-e99f-4f36-a31b-2951697e2bb6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:08:40.003422 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6073f9d2-e99f-4f36-a31b-2951697e2bb6 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:08:40.015512 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1141/2305] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:08:39 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:08:41.117204 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9ad003-a204-437e-bb93-fd7b067df227 service nova] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:08:41.126742 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9ad003-a204-437e-bb93-fd7b067df227 service nova] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:08:41.130430 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c73ecf4-917f-4b46-8555-dafaa733259b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:08:41.131020 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c73ecf4-917f-4b46-8555-dafaa733259b service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:08:41.131304 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c73ecf4-917f-4b46-8555-dafaa733259b service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:08:41.131578 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c73ecf4-917f-4b46-8555-dafaa733259b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:08:41.141850 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c73ecf4-917f-4b46-8555-dafaa733259b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:08:41.143271 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c73ecf4-917f-4b46-8555-dafaa733259b service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:08:41.154821 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1165/2306] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:08:41 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:08:50.010565 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=51 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:08:51.150104 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=12 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:09:11.330580 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6073f9d2-e99f-4f36-a31b-2951697e2bb6 service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:09:11.340147 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6073f9d2-e99f-4f36-a31b-2951697e2bb6 service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:09:11.343784 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b8a151-5576-46d3-a169-73d09b756d4f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:09:11.344372 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b8a151-5576-46d3-a169-73d09b756d4f service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:09:11.344678 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b8a151-5576-46d3-a169-73d09b756d4f service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:09:11.345031 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b8a151-5576-46d3-a169-73d09b756d4f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:09:11.352972 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b8a151-5576-46d3-a169-73d09b756d4f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:09:11.354396 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b8a151-5576-46d3-a169-73d09b756d4f service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:09:11.362286 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1142/2307] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:09:11 2022] GET /identity/v3/auth/tokens => generated 3131 bytes in 32 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:09:21.358952 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=6 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:09:50.157728 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273b43c9-2984-45c8-9e8a-1f7ca17a6a02 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:09:50.158408 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273b43c9-2984-45c8-9e8a-1f7ca17a6a02 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:09:50.158408 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273b43c9-2984-45c8-9e8a-1f7ca17a6a02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:09:50.196756 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-273b43c9-2984-45c8-9e8a-1f7ca17a6a02 None None] MFA Rules not processed for user `c058b07c1e9e443ebacbf38b1f13e58b`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:09:50.197959 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-273b43c9-2984-45c8-9e8a-1f7ca17a6a02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:09:50.215111 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1166/2308] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:09:50 2022] POST /identity/v3/auth/tokens => generated 3274 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:09:50.527815 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b8a151-5576-46d3-a169-73d09b756d4f service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:09:50.537540 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b8a151-5576-46d3-a169-73d09b756d4f service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:09:50.542763 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e6320b-d2cd-4daa-855a-b95b5f69fb3f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:09:50.543465 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e6320b-d2cd-4daa-855a-b95b5f69fb3f service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:09:50.543866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e6320b-d2cd-4daa-855a-b95b5f69fb3f service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:09:50.544240 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e6320b-d2cd-4daa-855a-b95b5f69fb3f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:09:50.546421 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-44e6320b-d2cd-4daa-855a-b95b5f69fb3f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:09:50.558780 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e6320b-d2cd-4daa-855a-b95b5f69fb3f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:09:50.560349 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e6320b-d2cd-4daa-855a-b95b5f69fb3f service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:09:50.569569 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1143/2309] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:09:50 2022] GET /identity/v3/auth/tokens => generated 3131 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:09:51.024968 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273b43c9-2984-45c8-9e8a-1f7ca17a6a02 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:09:51.037332 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273b43c9-2984-45c8-9e8a-1f7ca17a6a02 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:09:51.043246 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e5408a9-8063-4911-be13-d5492031eff1 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0ee0002a76c3471f819b37fef510ef99', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:09:51.043888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5408a9-8063-4911-be13-d5492031eff1 service placement] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:09:51.044169 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5408a9-8063-4911-be13-d5492031eff1 service placement] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:09:51.044440 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5408a9-8063-4911-be13-d5492031eff1 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:09:51.046277 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-3e5408a9-8063-4911-be13-d5492031eff1 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:09:51.058511 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e5408a9-8063-4911-be13-d5492031eff1 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:09:51.062073 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e5408a9-8063-4911-be13-d5492031eff1 service placement] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:09:51.070988 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1167/2310] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:09:51 2022] GET /identity/v3/auth/tokens => generated 3318 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:09:53.581875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e6320b-d2cd-4daa-855a-b95b5f69fb3f service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:09:53.597893 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e6320b-d2cd-4daa-855a-b95b5f69fb3f service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:09:53.604413 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737ec17e-cc5c-4335-84fe-af892f1fb44d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:09:53.604935 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737ec17e-cc5c-4335-84fe-af892f1fb44d service nova] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:09:53.605216 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737ec17e-cc5c-4335-84fe-af892f1fb44d service nova] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:09:53.605537 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737ec17e-cc5c-4335-84fe-af892f1fb44d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:09:53.607449 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-737ec17e-cc5c-4335-84fe-af892f1fb44d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:09:53.620254 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-737ec17e-cc5c-4335-84fe-af892f1fb44d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:09:53.621883 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-737ec17e-cc5c-4335-84fe-af892f1fb44d service nova] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:09:53.634837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1144/2311] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:09:53 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:09:58.159328 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e5408a9-8063-4911-be13-d5492031eff1 service placement] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:09:58.173026 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e5408a9-8063-4911-be13-d5492031eff1 service placement] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:09:58.179301 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d76027-c64c-45fc-9543-2e1185c3867b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:09:58.180038 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d76027-c64c-45fc-9543-2e1185c3867b service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:09:58.180408 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d76027-c64c-45fc-9543-2e1185c3867b service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:09:58.180747 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d76027-c64c-45fc-9543-2e1185c3867b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:09:58.182699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-07d76027-c64c-45fc-9543-2e1185c3867b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:09:58.196773 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d76027-c64c-45fc-9543-2e1185c3867b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:09:58.198812 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d76027-c64c-45fc-9543-2e1185c3867b service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:09:58.212016 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1168/2312] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:09:58 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:10:03.628073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=12 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:10:08.205707 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=26 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:10:49.664468 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737ec17e-cc5c-4335-84fe-af892f1fb44d service nova] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:10:49.677118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737ec17e-cc5c-4335-84fe-af892f1fb44d service nova] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:10:49.682690 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d517d05-ee91-421c-8304-a78fbdd8c86b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:10:49.683931 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d517d05-ee91-421c-8304-a78fbdd8c86b service nova] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:10:49.684227 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d517d05-ee91-421c-8304-a78fbdd8c86b service nova] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:10:49.685337 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d517d05-ee91-421c-8304-a78fbdd8c86b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:10:49.687280 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-2d517d05-ee91-421c-8304-a78fbdd8c86b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:10:49.721821 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d517d05-ee91-421c-8304-a78fbdd8c86b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:10:49.723430 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d517d05-ee91-421c-8304-a78fbdd8c86b service nova] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:10:49.742664 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1145/2313] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:10:49 2022] GET /identity/v3/auth/tokens => generated 3272 bytes in 79 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:10:59.740003 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=19 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:11:09.645549 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d76027-c64c-45fc-9543-2e1185c3867b service nova] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:11:09.647162 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-07d76027-c64c-45fc-9543-2e1185c3867b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:11:09.661088 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d76027-c64c-45fc-9543-2e1185c3867b service nova] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:11:09.666022 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff0ac31-987b-4387-a779-71dc1d81b124 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0ee0002a76c3471f819b37fef510ef99', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:11:09.666626 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff0ac31-987b-4387-a779-71dc1d81b124 service placement] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:11:09.666921 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff0ac31-987b-4387-a779-71dc1d81b124 service placement] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:11:09.667709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff0ac31-987b-4387-a779-71dc1d81b124 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:11:09.683759 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff0ac31-987b-4387-a779-71dc1d81b124 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:11:09.685511 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff0ac31-987b-4387-a779-71dc1d81b124 service placement] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:11:09.696965 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1169/2314] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:11:09 2022] GET /identity/v3/auth/tokens => generated 3318 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:11:19.691812 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=6 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:11:52.552852 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d517d05-ee91-421c-8304-a78fbdd8c86b service nova] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:11:52.570375 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d517d05-ee91-421c-8304-a78fbdd8c86b service nova] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:11:52.578873 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8833fe0-6183-4889-a578-51f2bb5c15f7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:11:52.582366 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8833fe0-6183-4889-a578-51f2bb5c15f7 service nova] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:11:52.583114 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8833fe0-6183-4889-a578-51f2bb5c15f7 service nova] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:11:52.584141 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8833fe0-6183-4889-a578-51f2bb5c15f7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:11:52.587743 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-c8833fe0-6183-4889-a578-51f2bb5c15f7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:11:52.620852 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8833fe0-6183-4889-a578-51f2bb5c15f7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:11:52.622738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8833fe0-6183-4889-a578-51f2bb5c15f7 service nova] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:11:52.638090 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1146/2315] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:11:52 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:11:54.223852 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4835d2b4-bad3-4c19-b142-a844a0d112df None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:11:54.224404 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4835d2b4-bad3-4c19-b142-a844a0d112df None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:11:54.224404 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4835d2b4-bad3-4c19-b142-a844a0d112df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:11:54.252513 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-4835d2b4-bad3-4c19-b142-a844a0d112df None None] MFA Rules not processed for user `ee49fabdcd304c72afefa48a25644bae`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:11:54.253733 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-4835d2b4-bad3-4c19-b142-a844a0d112df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:11:54.260620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1170/2316] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:11:54 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:11:54.579804 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8833fe0-6183-4889-a578-51f2bb5c15f7 service nova] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:11:54.589119 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8833fe0-6183-4889-a578-51f2bb5c15f7 service nova] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:11:54.592654 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d43432b-1329-4149-a92a-c29f96aa96cc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:11:54.593237 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d43432b-1329-4149-a92a-c29f96aa96cc service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:11:54.593512 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d43432b-1329-4149-a92a-c29f96aa96cc service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:11:54.593779 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d43432b-1329-4149-a92a-c29f96aa96cc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:11:54.595449 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-7d43432b-1329-4149-a92a-c29f96aa96cc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:11:54.604740 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d43432b-1329-4149-a92a-c29f96aa96cc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:11:54.606145 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d43432b-1329-4149-a92a-c29f96aa96cc service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:11:54.613899 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1147/2317] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:11:54 2022] GET /identity/v3/auth/tokens => generated 3131 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:12:04.250095 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=6 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:12:04.610463 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=12 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:12:44.928328 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2aff013-639b-4e9f-be57-5d0eab462eeb None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:12:44.928831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2aff013-639b-4e9f-be57-5d0eab462eeb None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:12:44.928921 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2aff013-639b-4e9f-be57-5d0eab462eeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:12:44.991545 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-f2aff013-639b-4e9f-be57-5d0eab462eeb None None] MFA Rules not processed for user `a6c17620ead14cb49d962990e1b1e0d8`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:12:44.992710 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-f2aff013-639b-4e9f-be57-5d0eab462eeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:12:45.054434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d43432b-1329-4149-a92a-c29f96aa96cc service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:12:45.056485 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-7d43432b-1329-4149-a92a-c29f96aa96cc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:12:45.060123 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1171/2318] 10.222.0.102 () {58 vars in 972 bytes} [Wed Aug 24 16:12:44 2022] POST /identity/v3/auth/tokens => generated 3205 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:12:45.081909 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d43432b-1329-4149-a92a-c29f96aa96cc service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:12:45.085887 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72e4a91-9d9f-4b98-8452-3533f5e5c286 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:12:45.086484 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72e4a91-9d9f-4b98-8452-3533f5e5c286 service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:12:45.086759 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72e4a91-9d9f-4b98-8452-3533f5e5c286 service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:12:45.087021 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72e4a91-9d9f-4b98-8452-3533f5e5c286 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:12:45.096602 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72e4a91-9d9f-4b98-8452-3533f5e5c286 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:12:45.098048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72e4a91-9d9f-4b98-8452-3533f5e5c286 service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:12:45.102911 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2aff013-639b-4e9f-be57-5d0eab462eeb None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:12:45.104496 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-f2aff013-639b-4e9f-be57-5d0eab462eeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:12:45.107891 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1148/2319] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:12:45 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 16:12:45.203602 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2aff013-639b-4e9f-be57-5d0eab462eeb None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:12:45.209164 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f89955f-f1cf-48a1-ad7b-c284441ff57a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a070d4fd8c4466c857bd03965c11349', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:12:45.209768 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f89955f-f1cf-48a1-ad7b-c284441ff57a service swift] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:12:45.210125 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f89955f-f1cf-48a1-ad7b-c284441ff57a service swift] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:12:45.210309 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f89955f-f1cf-48a1-ad7b-c284441ff57a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:12:45.218803 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f89955f-f1cf-48a1-ad7b-c284441ff57a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:12:45.220902 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f89955f-f1cf-48a1-ad7b-c284441ff57a service swift] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:12:45.227505 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72e4a91-9d9f-4b98-8452-3533f5e5c286 service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:12:45.230539 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1172/2320] 10.222.0.102 () {60 vars in 1389 bytes} [Wed Aug 24 16:12:45 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 24 16:12:45.237808 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72e4a91-9d9f-4b98-8452-3533f5e5c286 service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:12:45.241645 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acfd8fe0-fdf4-4845-ae30-a46d58cdacd0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:12:45.242261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfd8fe0-fdf4-4845-ae30-a46d58cdacd0 service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:12:45.242542 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfd8fe0-fdf4-4845-ae30-a46d58cdacd0 service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:12:45.242932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfd8fe0-fdf4-4845-ae30-a46d58cdacd0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:12:45.252223 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acfd8fe0-fdf4-4845-ae30-a46d58cdacd0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:12:45.253733 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acfd8fe0-fdf4-4845-ae30-a46d58cdacd0 service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:12:45.286700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1149/2321] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:12:45 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 16:12:45.804790 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f89955f-f1cf-48a1-ad7b-c284441ff57a service swift] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:12:45.815105 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f89955f-f1cf-48a1-ad7b-c284441ff57a service swift] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:12:45.818914 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe50e61f-69a6-4ae8-986d-235c8483be31 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:12:45.819529 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe50e61f-69a6-4ae8-986d-235c8483be31 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:12:45.819848 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe50e61f-69a6-4ae8-986d-235c8483be31 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:12:45.820126 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe50e61f-69a6-4ae8-986d-235c8483be31 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:12:45.824802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-fe50e61f-69a6-4ae8-986d-235c8483be31 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:12:45.859753 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:12:45.915926 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe50e61f-69a6-4ae8-986d-235c8483be31 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:12:45.917405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe50e61f-69a6-4ae8-986d-235c8483be31 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:12:45.937367 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1173/2322] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:12:45 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:12:55.283526 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=24 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:12:55.934724 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=28 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:13:50.680618 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acfd8fe0-fdf4-4845-ae30-a46d58cdacd0 service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:13:50.753054 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acfd8fe0-fdf4-4845-ae30-a46d58cdacd0 service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:13:50.759073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7738cb3-06d1-47cd-a76b-2c787ee6354c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:13:50.759699 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7738cb3-06d1-47cd-a76b-2c787ee6354c service nova] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:13:50.759986 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7738cb3-06d1-47cd-a76b-2c787ee6354c service nova] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:13:50.760262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7738cb3-06d1-47cd-a76b-2c787ee6354c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:13:50.768200 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7738cb3-06d1-47cd-a76b-2c787ee6354c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:13:50.769809 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7738cb3-06d1-47cd-a76b-2c787ee6354c service nova] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:13:50.779745 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1150/2323] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:13:50 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 16:13:50.964064 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe50e61f-69a6-4ae8-986d-235c8483be31 service neutron] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:13:50.965709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-fe50e61f-69a6-4ae8-986d-235c8483be31 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:13:51.034270 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe50e61f-69a6-4ae8-986d-235c8483be31 service neutron] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:13:51.039261 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af52e58-b4ea-4177-a051-c037a306a4bd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:13:51.039907 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af52e58-b4ea-4177-a051-c037a306a4bd service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:13:51.040221 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af52e58-b4ea-4177-a051-c037a306a4bd service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:13:51.040501 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af52e58-b4ea-4177-a051-c037a306a4bd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:13:51.042341 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-9af52e58-b4ea-4177-a051-c037a306a4bd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:13:51.068350 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af52e58-b4ea-4177-a051-c037a306a4bd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:13:51.069780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af52e58-b4ea-4177-a051-c037a306a4bd service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:13:51.080461 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1174/2324] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:13:50 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:13:51.179788 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7738cb3-06d1-47cd-a76b-2c787ee6354c service nova] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:13:51.181382 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-f7738cb3-06d1-47cd-a76b-2c787ee6354c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:13:51.242003 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7738cb3-06d1-47cd-a76b-2c787ee6354c service nova] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:13:51.245670 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108b5862-5ab7-4373-a9b5-1d6b18f3824e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:13:51.246268 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108b5862-5ab7-4373-a9b5-1d6b18f3824e service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:13:51.246545 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108b5862-5ab7-4373-a9b5-1d6b18f3824e service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:13:51.246811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108b5862-5ab7-4373-a9b5-1d6b18f3824e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:13:51.248648 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-108b5862-5ab7-4373-a9b5-1d6b18f3824e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:13:51.334093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-108b5862-5ab7-4373-a9b5-1d6b18f3824e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:13:51.335838 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-108b5862-5ab7-4373-a9b5-1d6b18f3824e service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:13:51.354066 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1151/2325] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:13:51 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:13:51.397905 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af52e58-b4ea-4177-a051-c037a306a4bd service neutron] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:13:51.407344 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af52e58-b4ea-4177-a051-c037a306a4bd service neutron] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:13:51.411532 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb43ced-1aa1-427b-8676-65401b1631f5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:13:51.412179 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb43ced-1aa1-427b-8676-65401b1631f5 service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:13:51.412462 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb43ced-1aa1-427b-8676-65401b1631f5 service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:13:51.412737 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb43ced-1aa1-427b-8676-65401b1631f5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:13:51.421525 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb43ced-1aa1-427b-8676-65401b1631f5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:13:51.423006 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb43ced-1aa1-427b-8676-65401b1631f5 service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:13:51.433948 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1175/2326] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:13:51 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:13:52.262877 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108b5862-5ab7-4373-a9b5-1d6b18f3824e service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:13:52.272129 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108b5862-5ab7-4373-a9b5-1d6b18f3824e service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:13:52.275661 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38412bb-1528-477d-954d-dd7534300b4f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:13:52.276243 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38412bb-1528-477d-954d-dd7534300b4f service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:13:52.276518 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38412bb-1528-477d-954d-dd7534300b4f service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:13:52.276787 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38412bb-1528-477d-954d-dd7534300b4f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:13:52.278586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-f38412bb-1528-477d-954d-dd7534300b4f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:13:52.292202 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f38412bb-1528-477d-954d-dd7534300b4f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:13:52.293626 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f38412bb-1528-477d-954d-dd7534300b4f service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:13:52.298235 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:13:52.306212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1152/2327] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:13:52 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:13:56.473525 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ead6a47-7cfc-46f8-9664-3f75d2e957d3 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:13:56.474014 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ead6a47-7cfc-46f8-9664-3f75d2e957d3 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:13:56.474098 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ead6a47-7cfc-46f8-9664-3f75d2e957d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:13:56.493517 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-1ead6a47-7cfc-46f8-9664-3f75d2e957d3 None None] MFA Rules not processed for user `ee49fabdcd304c72afefa48a25644bae`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:13:56.494708 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-1ead6a47-7cfc-46f8-9664-3f75d2e957d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:13:56.501461 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1176/2328] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:13:56 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:14:02.301524 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:14:06.487604 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=45 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:14:17.304829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38412bb-1528-477d-954d-dd7534300b4f service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:14:17.306415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-f38412bb-1528-477d-954d-dd7534300b4f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:14:17.317205 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38412bb-1528-477d-954d-dd7534300b4f service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:14:17.322164 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f133116-d763-452a-9475-cb81542d0003 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:14:17.322781 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f133116-d763-452a-9475-cb81542d0003 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:14:17.323064 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f133116-d763-452a-9475-cb81542d0003 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:14:17.323367 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f133116-d763-452a-9475-cb81542d0003 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:14:17.325333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-5f133116-d763-452a-9475-cb81542d0003 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:14:17.335830 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f133116-d763-452a-9475-cb81542d0003 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:14:17.337545 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f133116-d763-452a-9475-cb81542d0003 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:14:17.345545 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1153/2329] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:14:17 2022] GET /identity/v3/auth/tokens => generated 3131 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:14:27.342036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=6 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:14:46.223954 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ead6a47-7cfc-46f8-9664-3f75d2e957d3 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:14:46.235407 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ead6a47-7cfc-46f8-9664-3f75d2e957d3 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:14:46.239768 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92337b62-82bb-4518-b626-11d838d8a8f7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:14:46.240654 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92337b62-82bb-4518-b626-11d838d8a8f7 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:14:46.241205 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92337b62-82bb-4518-b626-11d838d8a8f7 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:14:46.241564 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92337b62-82bb-4518-b626-11d838d8a8f7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:14:46.243435 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-92337b62-82bb-4518-b626-11d838d8a8f7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:14:46.257695 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92337b62-82bb-4518-b626-11d838d8a8f7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:14:46.259147 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92337b62-82bb-4518-b626-11d838d8a8f7 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:14:46.269470 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1177/2330] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:14:46 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:14:49.392263 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f133116-d763-452a-9475-cb81542d0003 service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:14:49.404775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f133116-d763-452a-9475-cb81542d0003 service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:14:49.410028 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f8dbea-e26f-4eea-aea9-97b542cb2264 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0ee0002a76c3471f819b37fef510ef99', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:14:49.410632 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f8dbea-e26f-4eea-aea9-97b542cb2264 service placement] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:14:49.410946 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f8dbea-e26f-4eea-aea9-97b542cb2264 service placement] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:14:49.411248 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f8dbea-e26f-4eea-aea9-97b542cb2264 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:14:49.414337 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-04f8dbea-e26f-4eea-aea9-97b542cb2264 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:14:49.426338 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f8dbea-e26f-4eea-aea9-97b542cb2264 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:14:49.427788 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f8dbea-e26f-4eea-aea9-97b542cb2264 service placement] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:14:49.443567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1154/2331] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:14:49 2022] GET /identity/v3/auth/tokens => generated 3318 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:14:49.449861 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92337b62-82bb-4518-b626-11d838d8a8f7 service neutron] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:14:49.451496 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-92337b62-82bb-4518-b626-11d838d8a8f7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:14:49.474497 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92337b62-82bb-4518-b626-11d838d8a8f7 service neutron] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:14:49.481119 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30589f57-6c3f-4119-ba16-2c4c957de60f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:14:49.481772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30589f57-6c3f-4119-ba16-2c4c957de60f admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:14:49.482056 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30589f57-6c3f-4119-ba16-2c4c957de60f admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:14:49.482361 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30589f57-6c3f-4119-ba16-2c4c957de60f admin admin] PATH_INFO: `/v3/users/38f7d31cf0f541f58af7d3d850af6472` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:14:49.483573 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30589f57-6c3f-4119-ba16-2c4c957de60f admin admin] RBAC: Authorizing `identity:delete_user(user_id=38f7d31cf0f541f58af7d3d850af6472)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:14:49.485535 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30589f57-6c3f-4119-ba16-2c4c957de60f admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:14:49.531260 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-30589f57-6c3f-4119-ba16-2c4c957de60f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '38f7d31cf0f541f58af7d3d850af6472'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:14:49.531933 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-30589f57-6c3f-4119-ba16-2c4c957de60f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '38f7d31cf0f541f58af7d3d850af6472'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:14:49.532527 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-30589f57-6c3f-4119-ba16-2c4c957de60f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '38f7d31cf0f541f58af7d3d850af6472'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:14:49.540395 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-30589f57-6c3f-4119-ba16-2c4c957de60f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '38f7d31cf0f541f58af7d3d850af6472'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:14:49.553235 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-30589f57-6c3f-4119-ba16-2c4c957de60f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '38f7d31cf0f541f58af7d3d850af6472'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:14:49.564785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1178/2332] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:14:49 2022] DELETE /identity/v3/users/38f7d31cf0f541f58af7d3d850af6472 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:14:49.740454 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f8dbea-e26f-4eea-aea9-97b542cb2264 service placement] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:14:49.751453 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f8dbea-e26f-4eea-aea9-97b542cb2264 service placement] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:14:49.813751 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baafb9aa-4e06-45fa-8c3b-cde84ee5466b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:14:49.814383 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baafb9aa-4e06-45fa-8c3b-cde84ee5466b admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:14:49.814668 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baafb9aa-4e06-45fa-8c3b-cde84ee5466b admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:14:49.814947 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baafb9aa-4e06-45fa-8c3b-cde84ee5466b admin admin] PATH_INFO: `/v3/projects/b3568d52e7dd45b58640c06eddc5ea44` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:14:49.816115 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baafb9aa-4e06-45fa-8c3b-cde84ee5466b admin admin] RBAC: Authorizing `identity:delete_project(project_id=b3568d52e7dd45b58640c06eddc5ea44)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:14:49.818049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baafb9aa-4e06-45fa-8c3b-cde84ee5466b admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:14:49.867742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-baafb9aa-4e06-45fa-8c3b-cde84ee5466b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b3568d52e7dd45b58640c06eddc5ea44', 'request_id': 'req-baafb9aa-4e06-45fa-8c3b-cde84ee5466b'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:14:49.874042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-baafb9aa-4e06-45fa-8c3b-cde84ee5466b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b3568d52e7dd45b58640c06eddc5ea44', 'request_id': 'req-baafb9aa-4e06-45fa-8c3b-cde84ee5466b'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:14:49.879427 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-baafb9aa-4e06-45fa-8c3b-cde84ee5466b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b3568d52e7dd45b58640c06eddc5ea44', 'request_id': 'req-baafb9aa-4e06-45fa-8c3b-cde84ee5466b'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:14:49.892790 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-baafb9aa-4e06-45fa-8c3b-cde84ee5466b admin admin] The token cache is being invalidate because project b3568d52e7dd45b58640c06eddc5ea44 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:14:49.893078 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-baafb9aa-4e06-45fa-8c3b-cde84ee5466b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:14:49.893703 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-baafb9aa-4e06-45fa-8c3b-cde84ee5466b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:14:49.895729 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1155/2333] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:14:49 2022] DELETE /identity/v3/projects/b3568d52e7dd45b58640c06eddc5ea44 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:14:49.902546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b61305e-1d5a-4b0f-aab8-8f32acc1581c None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:14:49.902838 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b61305e-1d5a-4b0f-aab8-8f32acc1581c None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:14:49.903123 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b61305e-1d5a-4b0f-aab8-8f32acc1581c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:14:49.967026 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-5b61305e-1d5a-4b0f-aab8-8f32acc1581c None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:14:49.968500 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-5b61305e-1d5a-4b0f-aab8-8f32acc1581c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:14:50.030179 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1179/2334] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:14:49 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:14:50.038350 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b43d172-8274-41b7-be01-d259586724cb None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:14:50.038754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b43d172-8274-41b7-be01-d259586724cb None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:14:50.039040 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b43d172-8274-41b7-be01-d259586724cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:14:50.060000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-4b43d172-8274-41b7-be01-d259586724cb None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:14:50.061227 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-4b43d172-8274-41b7-be01-d259586724cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:14:50.070367 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1156/2335] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:14:50 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:14:50.074765 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b61305e-1d5a-4b0f-aab8-8f32acc1581c None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:14:50.085160 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b61305e-1d5a-4b0f-aab8-8f32acc1581c None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:14:50.090059 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e142a60-e8d4-483d-839e-78a527459e57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:14:50.090656 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e142a60-e8d4-483d-839e-78a527459e57 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:14:50.090940 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e142a60-e8d4-483d-839e-78a527459e57 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:14:50.091217 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e142a60-e8d4-483d-839e-78a527459e57 admin admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:14:50.091906 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e142a60-e8d4-483d-839e-78a527459e57 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 16:14:50.092211 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e142a60-e8d4-483d-839e-78a527459e57 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:14:50.093850 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e142a60-e8d4-483d-839e-78a527459e57 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:14:50.105648 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1180/2336] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 16:14:50 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 16:14:50.110118 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b43d172-8274-41b7-be01-d259586724cb None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:14:50.120373 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b43d172-8274-41b7-be01-d259586724cb None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:14:50.125088 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e936e9-2684-4b77-a4bd-63cb4ff52123 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:14:50.125674 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e936e9-2684-4b77-a4bd-63cb4ff52123 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:14:50.125955 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e936e9-2684-4b77-a4bd-63cb4ff52123 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:14:50.126233 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e936e9-2684-4b77-a4bd-63cb4ff52123 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:14:50.126860 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e936e9-2684-4b77-a4bd-63cb4ff52123 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:14:50.128804 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e936e9-2684-4b77-a4bd-63cb4ff52123 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:14:50.166207 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1157/2337] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:14:50 2022] POST /identity/v3/projects => generated 366 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:14:50.170588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e142a60-e8d4-483d-839e-78a527459e57 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:14:50.181464 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e142a60-e8d4-483d-839e-78a527459e57 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:14:50.193222 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:14:50.244068 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e002709-25aa-4984-aad1-c7a04a2c4f54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:14:50.244597 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e002709-25aa-4984-aad1-c7a04a2c4f54 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:14:50.244874 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e002709-25aa-4984-aad1-c7a04a2c4f54 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:14:50.245145 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e002709-25aa-4984-aad1-c7a04a2c4f54 admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:14:50.245769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e002709-25aa-4984-aad1-c7a04a2c4f54 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:14:50.247745 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e002709-25aa-4984-aad1-c7a04a2c4f54 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:14:50.269619 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1181/2338] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:14:50 2022] POST /identity/v3/users => generated 284 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 16:14:50.274555 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e936e9-2684-4b77-a4bd-63cb4ff52123 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:14:50.284831 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e936e9-2684-4b77-a4bd-63cb4ff52123 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:14:50.290280 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb273c8-6139-4a7f-a710-bb657647831b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:14:50.290875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb273c8-6139-4a7f-a710-bb657647831b admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:14:50.291164 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb273c8-6139-4a7f-a710-bb657647831b admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:14:50.291441 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb273c8-6139-4a7f-a710-bb657647831b admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:14:50.292120 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb273c8-6139-4a7f-a710-bb657647831b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:14:50.293721 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb273c8-6139-4a7f-a710-bb657647831b admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:14:50.304969 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1158/2339] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:14:50 2022] GET /identity/v3/roles => generated 1427 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 16:14:50.309099 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e002709-25aa-4984-aad1-c7a04a2c4f54 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:14:50.319541 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e002709-25aa-4984-aad1-c7a04a2c4f54 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:14:50.324329 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07744150-a1f7-460e-8634-78b4aeab738b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:14:50.324917 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07744150-a1f7-460e-8634-78b4aeab738b admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:14:50.325192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07744150-a1f7-460e-8634-78b4aeab738b admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:14:50.325461 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07744150-a1f7-460e-8634-78b4aeab738b admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:14:50.326100 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07744150-a1f7-460e-8634-78b4aeab738b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:14:50.327737 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07744150-a1f7-460e-8634-78b4aeab738b admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:14:50.340709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1182/2340] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:14:50 2022] GET /identity/v3/roles => generated 1427 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:14:50.344822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb273c8-6139-4a7f-a710-bb657647831b admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:14:50.354940 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb273c8-6139-4a7f-a710-bb657647831b admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:14:50.359545 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c20a28-2b08-41e0-b5a8-d332aa2f2eb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:14:50.360218 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c20a28-2b08-41e0-b5a8-d332aa2f2eb8 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:14:50.360500 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c20a28-2b08-41e0-b5a8-d332aa2f2eb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:14:50.360773 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c20a28-2b08-41e0-b5a8-d332aa2f2eb8 admin admin] PATH_INFO: `/v3/projects/8e2d05faa1694f8797cf147554adf260/users/d6107d0f26ec4d29b7a118f53c68ed36/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:14:50.375880 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c20a28-2b08-41e0-b5a8-d332aa2f2eb8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8e2d05faa1694f8797cf147554adf260, user_id=d6107d0f26ec4d29b7a118f53c68ed36, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:14:50.378569 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c20a28-2b08-41e0-b5a8-d332aa2f2eb8 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:14:50.392518 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1159/2341] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:14:50 2022] PUT /identity/v3/projects/8e2d05faa1694f8797cf147554adf260/users/d6107d0f26ec4d29b7a118f53c68ed36/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:14:50.404558 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7addcabf-8e2e-443a-b0cd-7f6f8593f03b None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:14:50.404845 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7addcabf-8e2e-443a-b0cd-7f6f8593f03b None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:14:50.405122 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7addcabf-8e2e-443a-b0cd-7f6f8593f03b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:14:50.432286 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-7addcabf-8e2e-443a-b0cd-7f6f8593f03b None None] MFA Rules not processed for user `d6107d0f26ec4d29b7a118f53c68ed36`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:14:50.433516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-7addcabf-8e2e-443a-b0cd-7f6f8593f03b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:14:50.501287 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1183/2342] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:14:50 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:14:50.506043 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c20a28-2b08-41e0-b5a8-d332aa2f2eb8 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:14:50.512615 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:14:50.570479 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c20a28-2b08-41e0-b5a8-d332aa2f2eb8 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:14:50.575220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc44082-480c-463a-89db-997749529974 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:14:50.575848 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc44082-480c-463a-89db-997749529974 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:14:50.576110 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc44082-480c-463a-89db-997749529974 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:14:50.576382 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc44082-480c-463a-89db-997749529974 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:14:50.577002 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc44082-480c-463a-89db-997749529974 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:14:50.578897 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc44082-480c-463a-89db-997749529974 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:14:50.609633 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1160/2343] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:14:50 2022] POST /identity/v3/projects => generated 366 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:14:50.610696 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7addcabf-8e2e-443a-b0cd-7f6f8593f03b None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:14:50.622172 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7addcabf-8e2e-443a-b0cd-7f6f8593f03b None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:14:50.677464 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a9af0c-4f14-4d5a-911b-21208081d95f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:14:50.678076 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a9af0c-4f14-4d5a-911b-21208081d95f admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:14:50.679096 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a9af0c-4f14-4d5a-911b-21208081d95f admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:14:50.679354 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a9af0c-4f14-4d5a-911b-21208081d95f admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:14:50.680034 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a9af0c-4f14-4d5a-911b-21208081d95f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:14:50.681947 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a9af0c-4f14-4d5a-911b-21208081d95f admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:14:50.703001 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1184/2344] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:14:50 2022] POST /identity/v3/users => generated 290 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 16:14:50.706963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc44082-480c-463a-89db-997749529974 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:14:50.717438 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc44082-480c-463a-89db-997749529974 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:14:50.722022 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f25be8-a56a-4e7a-9b66-dba6742f9d2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:14:50.722614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f25be8-a56a-4e7a-9b66-dba6742f9d2d admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:14:50.722898 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f25be8-a56a-4e7a-9b66-dba6742f9d2d admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:14:50.723170 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f25be8-a56a-4e7a-9b66-dba6742f9d2d admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:14:50.723851 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f25be8-a56a-4e7a-9b66-dba6742f9d2d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:14:50.726104 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f25be8-a56a-4e7a-9b66-dba6742f9d2d admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:14:50.736405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1161/2345] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:14:50 2022] GET /identity/v3/roles => generated 1427 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:14:50.740669 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a9af0c-4f14-4d5a-911b-21208081d95f admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:14:50.751432 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a9af0c-4f14-4d5a-911b-21208081d95f admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:14:50.756045 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce099a1-aaa3-4b40-b3b3-8e5733cef908 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:14:50.756702 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce099a1-aaa3-4b40-b3b3-8e5733cef908 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:14:50.756983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce099a1-aaa3-4b40-b3b3-8e5733cef908 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:14:50.757258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce099a1-aaa3-4b40-b3b3-8e5733cef908 admin admin] PATH_INFO: `/v3/projects/42044f80b5714c068d9b260927ecbc66/users/afcaa00babfd45bea77eac290e8e3d27/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:14:50.768672 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:14:50.774854 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce099a1-aaa3-4b40-b3b3-8e5733cef908 admin admin] RBAC: Authorizing `identity:create_grant(project_id=42044f80b5714c068d9b260927ecbc66, user_id=afcaa00babfd45bea77eac290e8e3d27, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:14:50.777763 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce099a1-aaa3-4b40-b3b3-8e5733cef908 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:14:50.794207 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1185/2346] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:14:50 2022] PUT /identity/v3/projects/42044f80b5714c068d9b260927ecbc66/users/afcaa00babfd45bea77eac290e8e3d27/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:14:50.799314 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f25be8-a56a-4e7a-9b66-dba6742f9d2d admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:14:50.811199 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f25be8-a56a-4e7a-9b66-dba6742f9d2d admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:14:50.871093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5d1cca-8a13-4f62-992f-f2909c04a12a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:14:50.871747 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5d1cca-8a13-4f62-992f-f2909c04a12a admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:14:50.872030 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5d1cca-8a13-4f62-992f-f2909c04a12a admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:14:50.872306 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5d1cca-8a13-4f62-992f-f2909c04a12a admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:14:50.872956 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa5d1cca-8a13-4f62-992f-f2909c04a12a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:14:50.874557 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa5d1cca-8a13-4f62-992f-f2909c04a12a admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:14:50.884563 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1162/2347] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:14:50 2022] GET /identity/v3/roles => generated 1427 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:14:50.888747 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce099a1-aaa3-4b40-b3b3-8e5733cef908 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:14:50.900233 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce099a1-aaa3-4b40-b3b3-8e5733cef908 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:14:50.904822 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d5c413-e2bd-46c6-93e1-d352637a49c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:14:50.905471 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d5c413-e2bd-46c6-93e1-d352637a49c5 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:14:50.905758 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d5c413-e2bd-46c6-93e1-d352637a49c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:14:50.906025 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d5c413-e2bd-46c6-93e1-d352637a49c5 admin admin] PATH_INFO: `/v3/projects/42044f80b5714c068d9b260927ecbc66/users/afcaa00babfd45bea77eac290e8e3d27/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:14:50.907965 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d5c413-e2bd-46c6-93e1-d352637a49c5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=42044f80b5714c068d9b260927ecbc66, user_id=afcaa00babfd45bea77eac290e8e3d27, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:14:50.910605 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d5c413-e2bd-46c6-93e1-d352637a49c5 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:14:50.922476 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1186/2348] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:14:50 2022] PUT /identity/v3/projects/42044f80b5714c068d9b260927ecbc66/users/afcaa00babfd45bea77eac290e8e3d27/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:14:50.932748 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473339ce-653b-4cc5-af82-fa9f13defe67 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:14:50.933036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473339ce-653b-4cc5-af82-fa9f13defe67 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:14:50.933314 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473339ce-653b-4cc5-af82-fa9f13defe67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:14:50.963488 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-473339ce-653b-4cc5-af82-fa9f13defe67 None None] MFA Rules not processed for user `afcaa00babfd45bea77eac290e8e3d27`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:14:50.964731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-473339ce-653b-4cc5-af82-fa9f13defe67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:14:51.030728 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:14:51.034656 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1163/2349] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:14:50 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:15:00.918442 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=9,keystone:INSERT=2 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:15:51.109248 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d5c413-e2bd-46c6-93e1-d352637a49c5 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:15:51.170476 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d5c413-e2bd-46c6-93e1-d352637a49c5 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:15:51.175246 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88cb9fb2-65a1-4714-a224-ecb0337c3071 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:15:51.175898 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88cb9fb2-65a1-4714-a224-ecb0337c3071 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:15:51.176186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88cb9fb2-65a1-4714-a224-ecb0337c3071 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:15:51.176459 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88cb9fb2-65a1-4714-a224-ecb0337c3071 admin admin] PATH_INFO: `/v3/users/d6107d0f26ec4d29b7a118f53c68ed36` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:15:51.177707 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88cb9fb2-65a1-4714-a224-ecb0337c3071 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d6107d0f26ec4d29b7a118f53c68ed36)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:15:51.179656 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88cb9fb2-65a1-4714-a224-ecb0337c3071 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:15:51.227279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-88cb9fb2-65a1-4714-a224-ecb0337c3071 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd6107d0f26ec4d29b7a118f53c68ed36'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:15:51.228011 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-88cb9fb2-65a1-4714-a224-ecb0337c3071 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd6107d0f26ec4d29b7a118f53c68ed36'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:15:51.228622 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-88cb9fb2-65a1-4714-a224-ecb0337c3071 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd6107d0f26ec4d29b7a118f53c68ed36'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:15:51.236409 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-88cb9fb2-65a1-4714-a224-ecb0337c3071 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd6107d0f26ec4d29b7a118f53c68ed36'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:15:51.252222 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-88cb9fb2-65a1-4714-a224-ecb0337c3071 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd6107d0f26ec4d29b7a118f53c68ed36'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:15:51.266913 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1187/2350] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:15:51 2022] DELETE /identity/v3/users/d6107d0f26ec4d29b7a118f53c68ed36 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:15:51.271960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899bd352-3752-4b94-8309-82b848e18f72 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:15:51.272449 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899bd352-3752-4b94-8309-82b848e18f72 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:15:51.272547 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899bd352-3752-4b94-8309-82b848e18f72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:15:51.291666 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-899bd352-3752-4b94-8309-82b848e18f72 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:15:51.292834 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-899bd352-3752-4b94-8309-82b848e18f72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:15:51.352539 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1164/2351] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:15:51 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:15:51.363181 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88cb9fb2-65a1-4714-a224-ecb0337c3071 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:15:51.364551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-88cb9fb2-65a1-4714-a224-ecb0337c3071 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:15:51.422071 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88cb9fb2-65a1-4714-a224-ecb0337c3071 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:15:51.425923 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36190ec-74dd-4a14-b43b-f4dc5800efb4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:15:51.426527 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36190ec-74dd-4a14-b43b-f4dc5800efb4 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:15:51.426809 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36190ec-74dd-4a14-b43b-f4dc5800efb4 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:15:51.427109 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36190ec-74dd-4a14-b43b-f4dc5800efb4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:15:51.436459 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d36190ec-74dd-4a14-b43b-f4dc5800efb4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:15:51.437886 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d36190ec-74dd-4a14-b43b-f4dc5800efb4 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:15:51.447215 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1188/2352] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:15:51 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:15:51.707393 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899bd352-3752-4b94-8309-82b848e18f72 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:15:51.708829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-899bd352-3752-4b94-8309-82b848e18f72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:15:51.738457 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36190ec-74dd-4a14-b43b-f4dc5800efb4 service neutron] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:15:51.740171 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-d36190ec-74dd-4a14-b43b-f4dc5800efb4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:15:51.754218 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36190ec-74dd-4a14-b43b-f4dc5800efb4 service neutron] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:15:51.759064 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112d63f1-ccbc-47a5-b3d7-e1e981419da8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:15:51.759720 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112d63f1-ccbc-47a5-b3d7-e1e981419da8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:15:51.760002 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112d63f1-ccbc-47a5-b3d7-e1e981419da8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:15:51.760318 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112d63f1-ccbc-47a5-b3d7-e1e981419da8 admin admin] PATH_INFO: `/v3/projects/8e2d05faa1694f8797cf147554adf260` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:15:51.761405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112d63f1-ccbc-47a5-b3d7-e1e981419da8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8e2d05faa1694f8797cf147554adf260)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:15:51.763863 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112d63f1-ccbc-47a5-b3d7-e1e981419da8 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:15:51.782968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899bd352-3752-4b94-8309-82b848e18f72 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:15:51.785277 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:15:51.787535 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cabcee4-6741-47e1-8e69-30add8e56214 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:15:51.788245 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cabcee4-6741-47e1-8e69-30add8e56214 service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:15:51.788590 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cabcee4-6741-47e1-8e69-30add8e56214 service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:15:51.788933 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cabcee4-6741-47e1-8e69-30add8e56214 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:15:51.791242 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-1cabcee4-6741-47e1-8e69-30add8e56214 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:15:51.817554 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-112d63f1-ccbc-47a5-b3d7-e1e981419da8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8e2d05faa1694f8797cf147554adf260', 'request_id': 'req-112d63f1-ccbc-47a5-b3d7-e1e981419da8'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:15:51.826301 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-112d63f1-ccbc-47a5-b3d7-e1e981419da8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8e2d05faa1694f8797cf147554adf260', 'request_id': 'req-112d63f1-ccbc-47a5-b3d7-e1e981419da8'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:15:51.833952 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-112d63f1-ccbc-47a5-b3d7-e1e981419da8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8e2d05faa1694f8797cf147554adf260', 'request_id': 'req-112d63f1-ccbc-47a5-b3d7-e1e981419da8'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:15:51.849154 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-112d63f1-ccbc-47a5-b3d7-e1e981419da8 admin admin] The token cache is being invalidate because project 8e2d05faa1694f8797cf147554adf260 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:15:51.849154 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-112d63f1-ccbc-47a5-b3d7-e1e981419da8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:15:51.849687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-112d63f1-ccbc-47a5-b3d7-e1e981419da8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:15:51.851811 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1189/2353] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:15:51 2022] DELETE /identity/v3/projects/8e2d05faa1694f8797cf147554adf260 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:15:51.856014 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112d63f1-ccbc-47a5-b3d7-e1e981419da8 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:15:51.857451 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-112d63f1-ccbc-47a5-b3d7-e1e981419da8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:15:51.862962 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cabcee4-6741-47e1-8e69-30add8e56214 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:15:51.865041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cabcee4-6741-47e1-8e69-30add8e56214 service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:15:51.876322 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1165/2354] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:15:51 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 169 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:15:51.928038 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112d63f1-ccbc-47a5-b3d7-e1e981419da8 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:15:51.932819 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64eaa414-6ee4-4f63-a431-5c163fae8572 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:15:51.933458 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64eaa414-6ee4-4f63-a431-5c163fae8572 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:15:51.934365 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64eaa414-6ee4-4f63-a431-5c163fae8572 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:15:51.934634 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64eaa414-6ee4-4f63-a431-5c163fae8572 admin admin] PATH_INFO: `/v3/users/afcaa00babfd45bea77eac290e8e3d27` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:15:51.935829 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64eaa414-6ee4-4f63-a431-5c163fae8572 admin admin] RBAC: Authorizing `identity:delete_user(user_id=afcaa00babfd45bea77eac290e8e3d27)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:15:51.937757 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64eaa414-6ee4-4f63-a431-5c163fae8572 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:15:52.000102 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-64eaa414-6ee4-4f63-a431-5c163fae8572 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'afcaa00babfd45bea77eac290e8e3d27'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:15:52.000823 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-64eaa414-6ee4-4f63-a431-5c163fae8572 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'afcaa00babfd45bea77eac290e8e3d27'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:15:52.001495 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-64eaa414-6ee4-4f63-a431-5c163fae8572 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'afcaa00babfd45bea77eac290e8e3d27'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:15:52.009128 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-64eaa414-6ee4-4f63-a431-5c163fae8572 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'afcaa00babfd45bea77eac290e8e3d27'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:15:52.022233 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-64eaa414-6ee4-4f63-a431-5c163fae8572 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'afcaa00babfd45bea77eac290e8e3d27'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:15:52.036342 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1190/2355] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:15:51 2022] DELETE /identity/v3/users/afcaa00babfd45bea77eac290e8e3d27 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:15:52.317581 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cabcee4-6741-47e1-8e69-30add8e56214 service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:15:52.328706 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cabcee4-6741-47e1-8e69-30add8e56214 service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:15:52.381587 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:15:52.392025 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f886c22e-0270-462b-8895-21415696fd45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:15:52.392855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f886c22e-0270-462b-8895-21415696fd45 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:15:52.393292 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f886c22e-0270-462b-8895-21415696fd45 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:15:52.393843 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f886c22e-0270-462b-8895-21415696fd45 admin admin] PATH_INFO: `/v3/projects/42044f80b5714c068d9b260927ecbc66` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:15:52.395236 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f886c22e-0270-462b-8895-21415696fd45 admin admin] RBAC: Authorizing `identity:delete_project(project_id=42044f80b5714c068d9b260927ecbc66)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:15:52.397342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f886c22e-0270-462b-8895-21415696fd45 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:15:52.448033 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-f886c22e-0270-462b-8895-21415696fd45 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '42044f80b5714c068d9b260927ecbc66', 'request_id': 'req-f886c22e-0270-462b-8895-21415696fd45'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:15:52.454516 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-f886c22e-0270-462b-8895-21415696fd45 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '42044f80b5714c068d9b260927ecbc66', 'request_id': 'req-f886c22e-0270-462b-8895-21415696fd45'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:15:52.459960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-f886c22e-0270-462b-8895-21415696fd45 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '42044f80b5714c068d9b260927ecbc66', 'request_id': 'req-f886c22e-0270-462b-8895-21415696fd45'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:15:52.472925 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-f886c22e-0270-462b-8895-21415696fd45 admin admin] The token cache is being invalidate because project 42044f80b5714c068d9b260927ecbc66 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:15:52.473209 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-f886c22e-0270-462b-8895-21415696fd45 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:15:52.473827 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-f886c22e-0270-462b-8895-21415696fd45 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:15:52.475312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1166/2356] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:15:52 2022] DELETE /identity/v3/projects/42044f80b5714c068d9b260927ecbc66 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:15:52.480187 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4442d73a-252d-4b87-ab4b-bc736ad1151e None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:15:52.481314 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4442d73a-252d-4b87-ab4b-bc736ad1151e None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:15:52.481405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4442d73a-252d-4b87-ab4b-bc736ad1151e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:15:52.501499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-4442d73a-252d-4b87-ab4b-bc736ad1151e None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:15:52.502699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-4442d73a-252d-4b87-ab4b-bc736ad1151e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:15:52.524996 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:15:52.583453 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1191/2357] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:15:52 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:15:52.592442 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a28ff1-f367-4d3a-a7f0-a9a144f158b5 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:15:52.592757 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a28ff1-f367-4d3a-a7f0-a9a144f158b5 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:15:52.593035 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a28ff1-f367-4d3a-a7f0-a9a144f158b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:15:52.613296 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-39a28ff1-f367-4d3a-a7f0-a9a144f158b5 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:15:52.614550 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-39a28ff1-f367-4d3a-a7f0-a9a144f158b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:15:52.623219 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1167/2358] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:15:52 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:15:52.627707 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4442d73a-252d-4b87-ab4b-bc736ad1151e None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:15:52.640328 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4442d73a-252d-4b87-ab4b-bc736ad1151e None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:15:52.645355 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1eb4a4-6270-4112-993b-7540e063a7b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:15:52.645907 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1eb4a4-6270-4112-993b-7540e063a7b6 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:15:52.646195 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1eb4a4-6270-4112-993b-7540e063a7b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:15:52.646470 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1eb4a4-6270-4112-993b-7540e063a7b6 admin admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:15:52.647152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1eb4a4-6270-4112-993b-7540e063a7b6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 16:15:52.647490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1eb4a4-6270-4112-993b-7540e063a7b6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:15:52.649138 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1eb4a4-6270-4112-993b-7540e063a7b6 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:15:52.661797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1192/2359] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 16:15:52 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 16:15:52.666232 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a28ff1-f367-4d3a-a7f0-a9a144f158b5 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:15:52.675661 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a28ff1-f367-4d3a-a7f0-a9a144f158b5 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:15:52.680372 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d56d36-6139-4541-bb98-2359113dd0d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:15:52.681015 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d56d36-6139-4541-bb98-2359113dd0d2 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:15:52.681312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d56d36-6139-4541-bb98-2359113dd0d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:15:52.681564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d56d36-6139-4541-bb98-2359113dd0d2 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:15:52.682192 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d56d36-6139-4541-bb98-2359113dd0d2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:15:52.684216 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d56d36-6139-4541-bb98-2359113dd0d2 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:15:52.712042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1168/2360] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:15:52 2022] POST /identity/v3/projects => generated 410 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:15:52.717109 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1eb4a4-6270-4112-993b-7540e063a7b6 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:15:52.727317 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1eb4a4-6270-4112-993b-7540e063a7b6 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:15:52.788906 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2516a131-7cf1-4eee-a3ef-c48df9966f7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:15:52.789539 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2516a131-7cf1-4eee-a3ef-c48df9966f7b admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:15:52.789823 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2516a131-7cf1-4eee-a3ef-c48df9966f7b admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:15:52.790098 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2516a131-7cf1-4eee-a3ef-c48df9966f7b admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:15:52.790724 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2516a131-7cf1-4eee-a3ef-c48df9966f7b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:15:52.792695 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2516a131-7cf1-4eee-a3ef-c48df9966f7b admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:15:52.813827 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1193/2361] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:15:52 2022] POST /identity/v3/users => generated 306 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 16:15:52.818053 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d56d36-6139-4541-bb98-2359113dd0d2 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:15:52.829590 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d56d36-6139-4541-bb98-2359113dd0d2 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:15:52.834168 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ab8119-543a-4c4a-98da-699a914bee15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:15:52.834771 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ab8119-543a-4c4a-98da-699a914bee15 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:15:52.835056 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ab8119-543a-4c4a-98da-699a914bee15 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:15:52.835324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ab8119-543a-4c4a-98da-699a914bee15 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:15:52.836015 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ab8119-543a-4c4a-98da-699a914bee15 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:15:52.837612 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ab8119-543a-4c4a-98da-699a914bee15 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:15:52.858090 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1169/2362] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:15:52 2022] GET /identity/v3/roles => generated 1427 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:15:52.862339 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2516a131-7cf1-4eee-a3ef-c48df9966f7b admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:15:52.873188 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2516a131-7cf1-4eee-a3ef-c48df9966f7b admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:15:52.877994 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089b2f38-4789-4a90-830e-35117109aa7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:15:52.878574 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089b2f38-4789-4a90-830e-35117109aa7c admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:15:52.878851 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089b2f38-4789-4a90-830e-35117109aa7c admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:15:52.879188 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089b2f38-4789-4a90-830e-35117109aa7c admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:15:52.879816 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-089b2f38-4789-4a90-830e-35117109aa7c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:15:52.881423 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-089b2f38-4789-4a90-830e-35117109aa7c admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:15:52.896675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1194/2363] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:15:52 2022] GET /identity/v3/roles => generated 1427 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:15:52.900823 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ab8119-543a-4c4a-98da-699a914bee15 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:15:52.912674 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ab8119-543a-4c4a-98da-699a914bee15 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:15:52.917350 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f5c2a1-8819-440f-9b66-5e33c0180b3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:15:52.918005 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f5c2a1-8819-440f-9b66-5e33c0180b3d admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:15:52.918404 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f5c2a1-8819-440f-9b66-5e33c0180b3d admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:15:52.918871 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f5c2a1-8819-440f-9b66-5e33c0180b3d admin admin] PATH_INFO: `/v3/projects/5c80f94de05b42c5b9313332d137f4f1/users/b78a52fbc29b4baeab16157f56041e59/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:15:52.940083 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f5c2a1-8819-440f-9b66-5e33c0180b3d admin admin] RBAC: Authorizing `identity:create_grant(project_id=5c80f94de05b42c5b9313332d137f4f1, user_id=b78a52fbc29b4baeab16157f56041e59, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:15:52.945685 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f5c2a1-8819-440f-9b66-5e33c0180b3d admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:15:52.960389 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1170/2364] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:15:52 2022] PUT /identity/v3/projects/5c80f94de05b42c5b9313332d137f4f1/users/b78a52fbc29b4baeab16157f56041e59/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:15:52.972671 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988c4aab-0c79-4c55-83d5-90cbab80d49b None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:15:52.972962 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988c4aab-0c79-4c55-83d5-90cbab80d49b None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:15:52.973243 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988c4aab-0c79-4c55-83d5-90cbab80d49b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:15:53.001681 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-988c4aab-0c79-4c55-83d5-90cbab80d49b None None] MFA Rules not processed for user `b78a52fbc29b4baeab16157f56041e59`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:15:53.002916 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-988c4aab-0c79-4c55-83d5-90cbab80d49b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:15:53.072938 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1195/2365] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:15:52 2022] POST /identity/v3/auth/tokens => generated 3296 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:15:53.078110 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f5c2a1-8819-440f-9b66-5e33c0180b3d admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:15:53.139727 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f5c2a1-8819-440f-9b66-5e33c0180b3d admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:15:53.144406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ca514d-3dc4-4606-b1ed-7e74398ec78e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:15:53.145014 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ca514d-3dc4-4606-b1ed-7e74398ec78e admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:15:53.145292 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ca514d-3dc4-4606-b1ed-7e74398ec78e admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:15:53.145566 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ca514d-3dc4-4606-b1ed-7e74398ec78e admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:15:53.146204 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ca514d-3dc4-4606-b1ed-7e74398ec78e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:15:53.148198 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ca514d-3dc4-4606-b1ed-7e74398ec78e admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:15:53.178906 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1171/2366] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:15:53 2022] POST /identity/v3/projects => generated 408 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:15:53.184643 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988c4aab-0c79-4c55-83d5-90cbab80d49b None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:15:53.196817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988c4aab-0c79-4c55-83d5-90cbab80d49b None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:15:53.214269 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:15:53.257348 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870b4b4a-60c8-4273-ab35-e4c4beda7d06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:15:53.257961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870b4b4a-60c8-4273-ab35-e4c4beda7d06 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:15:53.258244 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870b4b4a-60c8-4273-ab35-e4c4beda7d06 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:15:53.258519 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870b4b4a-60c8-4273-ab35-e4c4beda7d06 admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:15:53.259148 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870b4b4a-60c8-4273-ab35-e4c4beda7d06 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:15:53.261130 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870b4b4a-60c8-4273-ab35-e4c4beda7d06 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:15:53.283720 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1196/2367] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:15:53 2022] POST /identity/v3/users => generated 311 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:15:53.288161 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ca514d-3dc4-4606-b1ed-7e74398ec78e admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:15:53.294320 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:15:53.300822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ca514d-3dc4-4606-b1ed-7e74398ec78e admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:15:53.306191 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1be4105-39e2-4c8d-a03a-48f5d9e37abb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:15:53.306900 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1be4105-39e2-4c8d-a03a-48f5d9e37abb admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:15:53.307184 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1be4105-39e2-4c8d-a03a-48f5d9e37abb admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:15:53.307456 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1be4105-39e2-4c8d-a03a-48f5d9e37abb admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:15:53.308136 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1be4105-39e2-4c8d-a03a-48f5d9e37abb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:15:53.310502 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1be4105-39e2-4c8d-a03a-48f5d9e37abb admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:15:53.322315 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1172/2368] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:15:53 2022] GET /identity/v3/roles => generated 1427 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 16:15:53.327271 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870b4b4a-60c8-4273-ab35-e4c4beda7d06 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:15:53.337462 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870b4b4a-60c8-4273-ab35-e4c4beda7d06 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:15:53.342437 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75ec52d-987e-4b4a-897b-e0e8b8dccac6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:15:53.343084 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75ec52d-987e-4b4a-897b-e0e8b8dccac6 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:15:53.343363 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75ec52d-987e-4b4a-897b-e0e8b8dccac6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:15:53.343662 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75ec52d-987e-4b4a-897b-e0e8b8dccac6 admin admin] PATH_INFO: `/v3/projects/b073c18a79a34b23a9fbf6149a3dea31/users/394b9bbbb9304cbea79f3c8537c0bf23/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:15:53.359570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75ec52d-987e-4b4a-897b-e0e8b8dccac6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b073c18a79a34b23a9fbf6149a3dea31, user_id=394b9bbbb9304cbea79f3c8537c0bf23, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:15:53.362316 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75ec52d-987e-4b4a-897b-e0e8b8dccac6 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:15:53.377115 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1197/2369] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:15:53 2022] PUT /identity/v3/projects/b073c18a79a34b23a9fbf6149a3dea31/users/394b9bbbb9304cbea79f3c8537c0bf23/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:15:53.381120 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1be4105-39e2-4c8d-a03a-48f5d9e37abb admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:15:53.391271 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1be4105-39e2-4c8d-a03a-48f5d9e37abb admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:15:53.446878 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbc1c47-a0a7-4bd2-9f0a-6e0e3bd782e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:15:53.447400 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbc1c47-a0a7-4bd2-9f0a-6e0e3bd782e2 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:15:53.447711 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbc1c47-a0a7-4bd2-9f0a-6e0e3bd782e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:15:53.447985 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbc1c47-a0a7-4bd2-9f0a-6e0e3bd782e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:15:53.448629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cbc1c47-a0a7-4bd2-9f0a-6e0e3bd782e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:15:53.450388 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cbc1c47-a0a7-4bd2-9f0a-6e0e3bd782e2 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:15:53.460241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1173/2370] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:15:53 2022] GET /identity/v3/roles => generated 1427 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 16:15:53.464702 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75ec52d-987e-4b4a-897b-e0e8b8dccac6 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:15:53.475324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75ec52d-987e-4b4a-897b-e0e8b8dccac6 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:15:53.479972 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed31c77-dd82-47f7-9008-23794ef7305c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:15:53.481129 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed31c77-dd82-47f7-9008-23794ef7305c admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:15:53.481667 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed31c77-dd82-47f7-9008-23794ef7305c admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:15:53.481667 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed31c77-dd82-47f7-9008-23794ef7305c admin admin] PATH_INFO: `/v3/projects/b073c18a79a34b23a9fbf6149a3dea31/users/394b9bbbb9304cbea79f3c8537c0bf23/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:15:53.483566 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed31c77-dd82-47f7-9008-23794ef7305c admin admin] RBAC: Authorizing `identity:create_grant(project_id=b073c18a79a34b23a9fbf6149a3dea31, user_id=394b9bbbb9304cbea79f3c8537c0bf23, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:15:53.486237 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed31c77-dd82-47f7-9008-23794ef7305c admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:15:53.497350 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1198/2371] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:15:53 2022] PUT /identity/v3/projects/b073c18a79a34b23a9fbf6149a3dea31/users/394b9bbbb9304cbea79f3c8537c0bf23/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:15:53.507574 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9299f86-f63e-4ac6-863c-fb1b425fffd6 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:15:53.507886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9299f86-f63e-4ac6-863c-fb1b425fffd6 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:15:53.508188 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9299f86-f63e-4ac6-863c-fb1b425fffd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:15:53.535364 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-e9299f86-f63e-4ac6-863c-fb1b425fffd6 None None] MFA Rules not processed for user `394b9bbbb9304cbea79f3c8537c0bf23`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:15:53.536675 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-e9299f86-f63e-4ac6-863c-fb1b425fffd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:15:53.607030 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1174/2372] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:15:53 2022] POST /identity/v3/auth/tokens => generated 3361 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:16:03.493031 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=35,keystone:INSERT=5 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:16:03.603568 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=67 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:16:17.643262 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed31c77-dd82-47f7-9008-23794ef7305c admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:16:17.645044 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-5ed31c77-dd82-47f7-9008-23794ef7305c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:16:17.762585 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed31c77-dd82-47f7-9008-23794ef7305c admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:16:17.768004 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93b0fe7-8cc9-4e4d-b72c-a9b003761dbb service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0ee0002a76c3471f819b37fef510ef99', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:16:17.768615 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93b0fe7-8cc9-4e4d-b72c-a9b003761dbb service placement] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:16:17.768897 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93b0fe7-8cc9-4e4d-b72c-a9b003761dbb service placement] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:16:17.769174 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93b0fe7-8cc9-4e4d-b72c-a9b003761dbb service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:16:17.770910 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-e93b0fe7-8cc9-4e4d-b72c-a9b003761dbb service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:16:17.782956 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93b0fe7-8cc9-4e4d-b72c-a9b003761dbb service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:16:17.784597 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93b0fe7-8cc9-4e4d-b72c-a9b003761dbb service placement] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:16:17.794518 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1199/2373] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:16:17 2022] GET /identity/v3/auth/tokens => generated 3318 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:16:27.790339 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=52 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:16:46.579644 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9299f86-f63e-4ac6-863c-fb1b425fffd6 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:16:46.581045 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-e9299f86-f63e-4ac6-863c-fb1b425fffd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:16:46.651030 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9299f86-f63e-4ac6-863c-fb1b425fffd6 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:16:46.654619 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b760e761-9f94-4309-9018-02bea4370bde service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:16:46.655270 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b760e761-9f94-4309-9018-02bea4370bde service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:16:46.655594 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b760e761-9f94-4309-9018-02bea4370bde service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:16:46.656069 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b760e761-9f94-4309-9018-02bea4370bde service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:16:46.665135 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b760e761-9f94-4309-9018-02bea4370bde service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:16:46.666557 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b760e761-9f94-4309-9018-02bea4370bde service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:16:46.676638 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1175/2374] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:16:46 2022] GET /identity/v3/auth/tokens => generated 3361 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:16:56.672623 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=30 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:17:52.481457 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93b0fe7-8cc9-4e4d-b72c-a9b003761dbb service placement] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:17:52.482960 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-e93b0fe7-8cc9-4e4d-b72c-a9b003761dbb service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:17:52.513637 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93b0fe7-8cc9-4e4d-b72c-a9b003761dbb service placement] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:17:52.517917 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74dc5bd0-d7e8-43c4-8ed5-de66254a5b16 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:17:52.518537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dc5bd0-d7e8-43c4-8ed5-de66254a5b16 service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:17:52.518810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dc5bd0-d7e8-43c4-8ed5-de66254a5b16 service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:17:52.519080 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dc5bd0-d7e8-43c4-8ed5-de66254a5b16 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:17:52.520811 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-74dc5bd0-d7e8-43c4-8ed5-de66254a5b16 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:17:52.604805 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74dc5bd0-d7e8-43c4-8ed5-de66254a5b16 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:17:52.606253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74dc5bd0-d7e8-43c4-8ed5-de66254a5b16 service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:17:52.616216 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1200/2375] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:17:52 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:17:52.664781 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b760e761-9f94-4309-9018-02bea4370bde service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:17:52.693339 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b760e761-9f94-4309-9018-02bea4370bde service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:17:52.697141 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-becf5cf4-23ac-4586-b1fc-cdb64d2d0955 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:17:52.697741 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-becf5cf4-23ac-4586-b1fc-cdb64d2d0955 service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:17:52.698024 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-becf5cf4-23ac-4586-b1fc-cdb64d2d0955 service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:17:52.698298 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-becf5cf4-23ac-4586-b1fc-cdb64d2d0955 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:17:52.700043 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-becf5cf4-23ac-4586-b1fc-cdb64d2d0955 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:17:52.772483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-becf5cf4-23ac-4586-b1fc-cdb64d2d0955 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:17:52.773906 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-becf5cf4-23ac-4586-b1fc-cdb64d2d0955 service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:17:52.784660 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1176/2376] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:17:52 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:17:54.480812 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74dc5bd0-d7e8-43c4-8ed5-de66254a5b16 service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:17:54.482340 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-74dc5bd0-d7e8-43c4-8ed5-de66254a5b16 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:17:54.600688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74dc5bd0-d7e8-43c4-8ed5-de66254a5b16 service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:17:54.608158 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e2a622-851d-46ae-ba20-f78853541e2a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:17:54.608927 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e2a622-851d-46ae-ba20-f78853541e2a service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:17:54.609279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e2a622-851d-46ae-ba20-f78853541e2a service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:17:54.609627 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e2a622-851d-46ae-ba20-f78853541e2a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:17:54.612766 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-b0e2a622-851d-46ae-ba20-f78853541e2a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:17:54.637152 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e2a622-851d-46ae-ba20-f78853541e2a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:17:54.638921 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e2a622-851d-46ae-ba20-f78853541e2a service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:17:54.651320 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1201/2377] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:17:54 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 171 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:18:02.779283 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=42 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:18:04.646222 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=89 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:18:46.799177 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-becf5cf4-23ac-4586-b1fc-cdb64d2d0955 service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:18:46.808158 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-becf5cf4-23ac-4586-b1fc-cdb64d2d0955 service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:18:46.811966 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8522f366-e862-4bfd-b0a7-da8506e6cf58 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:18:46.812567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8522f366-e862-4bfd-b0a7-da8506e6cf58 service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:18:46.812848 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8522f366-e862-4bfd-b0a7-da8506e6cf58 service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:18:46.813123 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8522f366-e862-4bfd-b0a7-da8506e6cf58 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:18:46.873159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8522f366-e862-4bfd-b0a7-da8506e6cf58 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:18:46.874600 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8522f366-e862-4bfd-b0a7-da8506e6cf58 service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:18:46.884199 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1177/2378] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:18:46 2022] GET /identity/v3/auth/tokens => generated 3296 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:18:56.880153 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=28 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:19:22.252829 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e2a622-851d-46ae-ba20-f78853541e2a service nova] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:19:22.254388 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-b0e2a622-851d-46ae-ba20-f78853541e2a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:19:22.347154 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e2a622-851d-46ae-ba20-f78853541e2a service nova] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:19:22.350769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67745fcb-21dc-4bfb-be27-571cf67673d5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:19:22.351374 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67745fcb-21dc-4bfb-be27-571cf67673d5 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:19:22.351674 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67745fcb-21dc-4bfb-be27-571cf67673d5 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:19:22.351948 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67745fcb-21dc-4bfb-be27-571cf67673d5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:19:22.353684 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-67745fcb-21dc-4bfb-be27-571cf67673d5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:19:22.363204 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67745fcb-21dc-4bfb-be27-571cf67673d5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:19:22.364647 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67745fcb-21dc-4bfb-be27-571cf67673d5 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:19:22.372442 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1202/2379] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:19:22 2022] GET /identity/v3/auth/tokens => generated 3131 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:19:32.368991 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=30 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:19:53.101318 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8522f366-e862-4bfd-b0a7-da8506e6cf58 service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:19:53.110163 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8522f366-e862-4bfd-b0a7-da8506e6cf58 service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:19:53.114461 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64bfa0a8-d74a-4d8c-8521-fa780a1a7e8f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:19:53.115084 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64bfa0a8-d74a-4d8c-8521-fa780a1a7e8f service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:19:53.115365 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64bfa0a8-d74a-4d8c-8521-fa780a1a7e8f service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:19:53.115726 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64bfa0a8-d74a-4d8c-8521-fa780a1a7e8f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:19:53.117415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-64bfa0a8-d74a-4d8c-8521-fa780a1a7e8f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:19:53.177794 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64bfa0a8-d74a-4d8c-8521-fa780a1a7e8f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:19:53.179459 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64bfa0a8-d74a-4d8c-8521-fa780a1a7e8f service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:19:53.190154 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1178/2380] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:19:53 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:19:53.381105 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67745fcb-21dc-4bfb-be27-571cf67673d5 service neutron] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:19:53.391909 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67745fcb-21dc-4bfb-be27-571cf67673d5 service neutron] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:19:53.397907 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8e5b24-149b-4b56-bcbe-cb8783099177 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:19:53.398656 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8e5b24-149b-4b56-bcbe-cb8783099177 service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:19:53.400814 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8e5b24-149b-4b56-bcbe-cb8783099177 service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:19:53.400814 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8e5b24-149b-4b56-bcbe-cb8783099177 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:19:53.401209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-4c8e5b24-149b-4b56-bcbe-cb8783099177 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:19:53.482806 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c8e5b24-149b-4b56-bcbe-cb8783099177 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:19:53.484279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c8e5b24-149b-4b56-bcbe-cb8783099177 service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:19:53.502390 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1203/2381] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:19:53 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:20:03.185228 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=28 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:20:03.499417 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=41 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:21:21.623092 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64bfa0a8-d74a-4d8c-8521-fa780a1a7e8f service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:21:21.635446 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64bfa0a8-d74a-4d8c-8521-fa780a1a7e8f service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:21:21.640891 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047f2c85-04cb-4a58-ab67-6deb9628514f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0ee0002a76c3471f819b37fef510ef99', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:21:21.641500 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047f2c85-04cb-4a58-ab67-6deb9628514f service placement] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:21:21.641783 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047f2c85-04cb-4a58-ab67-6deb9628514f service placement] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:21:21.642310 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047f2c85-04cb-4a58-ab67-6deb9628514f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:21:21.651888 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-047f2c85-04cb-4a58-ab67-6deb9628514f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:21:21.653670 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-047f2c85-04cb-4a58-ab67-6deb9628514f service placement] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:21:21.665146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1179/2382] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:21:21 2022] GET /identity/v3/auth/tokens => generated 3318 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:21:31.660677 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=6 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:21:54.278818 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8e5b24-149b-4b56-bcbe-cb8783099177 service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:21:54.280208 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-4c8e5b24-149b-4b56-bcbe-cb8783099177 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:21:54.292738 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8e5b24-149b-4b56-bcbe-cb8783099177 service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:21:54.296343 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3f8501-2d6a-4007-9705-59ed14332d0f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:21:54.296942 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3f8501-2d6a-4007-9705-59ed14332d0f service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:21:54.297224 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3f8501-2d6a-4007-9705-59ed14332d0f service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:21:54.297499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3f8501-2d6a-4007-9705-59ed14332d0f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:21:54.299192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-0f3f8501-2d6a-4007-9705-59ed14332d0f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:21:54.313121 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3f8501-2d6a-4007-9705-59ed14332d0f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:21:54.314544 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3f8501-2d6a-4007-9705-59ed14332d0f service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:21:54.325415 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1204/2383] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:21:54 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:21:55.535137 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047f2c85-04cb-4a58-ab67-6deb9628514f service placement] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:21:55.547424 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047f2c85-04cb-4a58-ab67-6deb9628514f service placement] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:21:55.551845 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5de826-c44c-4a94-b685-9efb19b5c560 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:21:55.553241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5de826-c44c-4a94-b685-9efb19b5c560 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:21:55.553716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5de826-c44c-4a94-b685-9efb19b5c560 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:21:55.554170 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5de826-c44c-4a94-b685-9efb19b5c560 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:21:55.556129 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-3b5de826-c44c-4a94-b685-9efb19b5c560 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:21:55.574474 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5de826-c44c-4a94-b685-9efb19b5c560 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:21:55.576180 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5de826-c44c-4a94-b685-9efb19b5c560 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:21:55.590438 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1180/2384] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:21:55 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:22:04.320676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=6 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:05.583369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=6 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:41.004297 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3f8501-2d6a-4007-9705-59ed14332d0f service neutron] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:41.014429 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3f8501-2d6a-4007-9705-59ed14332d0f service neutron] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:41.018171 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0839260e-2322-44bb-af4e-c43630acb17e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:41.018773 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0839260e-2322-44bb-af4e-c43630acb17e service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:41.019050 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0839260e-2322-44bb-af4e-c43630acb17e service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:41.019320 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0839260e-2322-44bb-af4e-c43630acb17e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:41.021078 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-0839260e-2322-44bb-af4e-c43630acb17e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:41.104966 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0839260e-2322-44bb-af4e-c43630acb17e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:41.106957 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0839260e-2322-44bb-af4e-c43630acb17e service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:41.126306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1205/2385] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:22:41 2022] GET /identity/v3/auth/tokens => generated 3272 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:22:42.028889 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5de826-c44c-4a94-b685-9efb19b5c560 service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:42.039677 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5de826-c44c-4a94-b685-9efb19b5c560 service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:42.043771 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1a7e2a-0a08-472f-b40f-ad5f4a213e71 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:42.044632 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1a7e2a-0a08-472f-b40f-ad5f4a213e71 service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:42.044915 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1a7e2a-0a08-472f-b40f-ad5f4a213e71 service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:42.045338 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1a7e2a-0a08-472f-b40f-ad5f4a213e71 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:42.056772 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb1a7e2a-0a08-472f-b40f-ad5f4a213e71 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:42.058209 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb1a7e2a-0a08-472f-b40f-ad5f4a213e71 service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:42.081173 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1181/2386] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:22:42 2022] GET /identity/v3/auth/tokens => generated 3361 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:22:42.084665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0839260e-2322-44bb-af4e-c43630acb17e service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:42.094983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0839260e-2322-44bb-af4e-c43630acb17e service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:42.099103 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75af4532-11a0-4621-80f8-692b0f867a1d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:42.100103 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75af4532-11a0-4621-80f8-692b0f867a1d service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:42.100103 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75af4532-11a0-4621-80f8-692b0f867a1d service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:42.100324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75af4532-11a0-4621-80f8-692b0f867a1d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:42.102262 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-75af4532-11a0-4621-80f8-692b0f867a1d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:42.192118 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75af4532-11a0-4621-80f8-692b0f867a1d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:42.193520 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75af4532-11a0-4621-80f8-692b0f867a1d service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:42.205434 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1206/2387] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:22:42 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:22:42.416120 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1a7e2a-0a08-472f-b40f-ad5f4a213e71 service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:42.437618 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1a7e2a-0a08-472f-b40f-ad5f4a213e71 service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:42.445845 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c880b7-5c66-42b7-909c-e9c226b47826 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:42.446489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c880b7-5c66-42b7-909c-e9c226b47826 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:42.446796 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c880b7-5c66-42b7-909c-e9c226b47826 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:42.447074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c880b7-5c66-42b7-909c-e9c226b47826 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:42.452227 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-b9c880b7-5c66-42b7-909c-e9c226b47826 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:42.498331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c880b7-5c66-42b7-909c-e9c226b47826 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:42.501105 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c880b7-5c66-42b7-909c-e9c226b47826 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:42.531297 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1182/2388] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:22:42 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:22:45.397999 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75af4532-11a0-4621-80f8-692b0f867a1d service neutron] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:45.409209 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75af4532-11a0-4621-80f8-692b0f867a1d service neutron] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:45.420902 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae0d24d-f694-46d2-8455-6043adfd79ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:45.421982 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae0d24d-f694-46d2-8455-6043adfd79ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:45.422466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae0d24d-f694-46d2-8455-6043adfd79ea admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:45.423022 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae0d24d-f694-46d2-8455-6043adfd79ea admin admin] PATH_INFO: `/v3/users/b78a52fbc29b4baeab16157f56041e59` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:45.424881 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ae0d24d-f694-46d2-8455-6043adfd79ea admin admin] RBAC: Authorizing `identity:delete_user(user_id=b78a52fbc29b4baeab16157f56041e59)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:45.427622 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ae0d24d-f694-46d2-8455-6043adfd79ea admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:45.519330 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-1ae0d24d-f694-46d2-8455-6043adfd79ea admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b78a52fbc29b4baeab16157f56041e59'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:45.520056 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-1ae0d24d-f694-46d2-8455-6043adfd79ea admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b78a52fbc29b4baeab16157f56041e59'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:45.520684 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-1ae0d24d-f694-46d2-8455-6043adfd79ea admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b78a52fbc29b4baeab16157f56041e59'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:45.532912 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-1ae0d24d-f694-46d2-8455-6043adfd79ea admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b78a52fbc29b4baeab16157f56041e59'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:45.541570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:45.559300 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-1ae0d24d-f694-46d2-8455-6043adfd79ea admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b78a52fbc29b4baeab16157f56041e59'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:45.578801 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1207/2389] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:22:45 2022] DELETE /identity/v3/users/b78a52fbc29b4baeab16157f56041e59 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:22:45.583230 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31167063-bb46-4191-b34b-cc60c11139e1 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:45.583742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31167063-bb46-4191-b34b-cc60c11139e1 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:45.584001 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31167063-bb46-4191-b34b-cc60c11139e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:45.619759 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-31167063-bb46-4191-b34b-cc60c11139e1 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:45.621296 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-31167063-bb46-4191-b34b-cc60c11139e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:45.721852 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1183/2390] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:45 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:22:45.732458 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae0d24d-f694-46d2-8455-6043adfd79ea admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:45.734279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-1ae0d24d-f694-46d2-8455-6043adfd79ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:45.810339 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae0d24d-f694-46d2-8455-6043adfd79ea admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:45.815211 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251c3542-81c1-4571-8984-8081b8e43d26 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:45.815936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251c3542-81c1-4571-8984-8081b8e43d26 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:45.816216 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251c3542-81c1-4571-8984-8081b8e43d26 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:45.816841 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251c3542-81c1-4571-8984-8081b8e43d26 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:45.831211 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251c3542-81c1-4571-8984-8081b8e43d26 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:45.832919 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251c3542-81c1-4571-8984-8081b8e43d26 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:45.851782 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1208/2391] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:22:45 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:22:46.616502 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31167063-bb46-4191-b34b-cc60c11139e1 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:46.619622 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-31167063-bb46-4191-b34b-cc60c11139e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:46.693528 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251c3542-81c1-4571-8984-8081b8e43d26 service neutron] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:46.694858 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-251c3542-81c1-4571-8984-8081b8e43d26 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:46.728590 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31167063-bb46-4191-b34b-cc60c11139e1 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:46.733989 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6db352-669b-4016-b0ec-054b38b14b40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:46.735526 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6db352-669b-4016-b0ec-054b38b14b40 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:46.736073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6db352-669b-4016-b0ec-054b38b14b40 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:46.736546 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6db352-669b-4016-b0ec-054b38b14b40 admin admin] PATH_INFO: `/v3/projects/5c80f94de05b42c5b9313332d137f4f1` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:46.738441 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe6db352-669b-4016-b0ec-054b38b14b40 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5c80f94de05b42c5b9313332d137f4f1)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:46.740612 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe6db352-669b-4016-b0ec-054b38b14b40 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:46.822463 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-fe6db352-669b-4016-b0ec-054b38b14b40 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5c80f94de05b42c5b9313332d137f4f1', 'request_id': 'req-fe6db352-669b-4016-b0ec-054b38b14b40'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:46.834325 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-fe6db352-669b-4016-b0ec-054b38b14b40 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5c80f94de05b42c5b9313332d137f4f1', 'request_id': 'req-fe6db352-669b-4016-b0ec-054b38b14b40'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:46.846982 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-fe6db352-669b-4016-b0ec-054b38b14b40 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5c80f94de05b42c5b9313332d137f4f1', 'request_id': 'req-fe6db352-669b-4016-b0ec-054b38b14b40'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:46.864662 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251c3542-81c1-4571-8984-8081b8e43d26 service neutron] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:46.868687 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-fe6db352-669b-4016-b0ec-054b38b14b40 admin admin] The token cache is being invalidate because project 5c80f94de05b42c5b9313332d137f4f1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:22:46.868986 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-fe6db352-669b-4016-b0ec-054b38b14b40 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:46.869878 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-fe6db352-669b-4016-b0ec-054b38b14b40 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:46.871602 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1184/2392] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:22:46 2022] DELETE /identity/v3/projects/5c80f94de05b42c5b9313332d137f4f1 => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:22:46.872876 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6db352-669b-4016-b0ec-054b38b14b40 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:46.875305 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-fe6db352-669b-4016-b0ec-054b38b14b40 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:46.917588 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:46.966696 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6db352-669b-4016-b0ec-054b38b14b40 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:46.971774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9b0285-f370-4cb4-9f18-00bd55663c42 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0ee0002a76c3471f819b37fef510ef99', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:46.972472 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9b0285-f370-4cb4-9f18-00bd55663c42 service placement] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:46.972740 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fba7c5-d12a-4418-ae9f-6e751830858b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:46.972869 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9b0285-f370-4cb4-9f18-00bd55663c42 service placement] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:46.973033 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9b0285-f370-4cb4-9f18-00bd55663c42 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:46.973427 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fba7c5-d12a-4418-ae9f-6e751830858b admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:46.973717 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fba7c5-d12a-4418-ae9f-6e751830858b admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:46.974015 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fba7c5-d12a-4418-ae9f-6e751830858b admin admin] PATH_INFO: `/v3/users/ee49fabdcd304c72afefa48a25644bae` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:46.974953 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-bb9b0285-f370-4cb4-9f18-00bd55663c42 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:46.975480 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1fba7c5-d12a-4418-ae9f-6e751830858b admin admin] RBAC: Authorizing `identity:delete_user(user_id=ee49fabdcd304c72afefa48a25644bae)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:46.982322 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1fba7c5-d12a-4418-ae9f-6e751830858b admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:46.987253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9b0285-f370-4cb4-9f18-00bd55663c42 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:46.988897 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9b0285-f370-4cb4-9f18-00bd55663c42 service placement] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:46.999838 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1209/2393] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:22:46 2022] GET /identity/v3/auth/tokens => generated 3318 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 16:22:47.008721 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9b0285-f370-4cb4-9f18-00bd55663c42 service placement] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:47.010325 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-bb9b0285-f370-4cb4-9f18-00bd55663c42 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:47.026272 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9b0285-f370-4cb4-9f18-00bd55663c42 service placement] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:47.033318 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2943746-2b92-4cd7-b333-e2ca015ad7b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:47.034024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2943746-2b92-4cd7-b333-e2ca015ad7b9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:47.034303 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2943746-2b92-4cd7-b333-e2ca015ad7b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:47.034578 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2943746-2b92-4cd7-b333-e2ca015ad7b9 admin admin] PATH_INFO: `/v3/users/394b9bbbb9304cbea79f3c8537c0bf23` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:47.035885 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2943746-2b92-4cd7-b333-e2ca015ad7b9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=394b9bbbb9304cbea79f3c8537c0bf23)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:47.037850 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2943746-2b92-4cd7-b333-e2ca015ad7b9 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:47.037967 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d1fba7c5-d12a-4418-ae9f-6e751830858b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ee49fabdcd304c72afefa48a25644bae'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:47.038589 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d1fba7c5-d12a-4418-ae9f-6e751830858b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ee49fabdcd304c72afefa48a25644bae'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:47.043289 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:DELETE=2,keystone:SELECT=98 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:47.047761 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d1fba7c5-d12a-4418-ae9f-6e751830858b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ee49fabdcd304c72afefa48a25644bae'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:47.067504 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d1fba7c5-d12a-4418-ae9f-6e751830858b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ee49fabdcd304c72afefa48a25644bae'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:47.068154 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d1fba7c5-d12a-4418-ae9f-6e751830858b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ee49fabdcd304c72afefa48a25644bae'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:47.083758 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1185/2394] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:22:46 2022] DELETE /identity/v3/users/ee49fabdcd304c72afefa48a25644bae => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:22:47.097001 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-a2943746-2b92-4cd7-b333-e2ca015ad7b9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '394b9bbbb9304cbea79f3c8537c0bf23'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:47.097804 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-a2943746-2b92-4cd7-b333-e2ca015ad7b9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '394b9bbbb9304cbea79f3c8537c0bf23'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:47.098481 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-a2943746-2b92-4cd7-b333-e2ca015ad7b9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '394b9bbbb9304cbea79f3c8537c0bf23'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:47.112114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-a2943746-2b92-4cd7-b333-e2ca015ad7b9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '394b9bbbb9304cbea79f3c8537c0bf23'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:47.128175 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-a2943746-2b92-4cd7-b333-e2ca015ad7b9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '394b9bbbb9304cbea79f3c8537c0bf23'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:47.150095 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1210/2395] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:22:47 2022] DELETE /identity/v3/users/394b9bbbb9304cbea79f3c8537c0bf23 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:22:47.315919 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fba7c5-d12a-4418-ae9f-6e751830858b admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:47.317751 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-d1fba7c5-d12a-4418-ae9f-6e751830858b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:47.425949 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fba7c5-d12a-4418-ae9f-6e751830858b admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:47.479434 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2943746-2b92-4cd7-b333-e2ca015ad7b9 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:47.481532 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-a2943746-2b92-4cd7-b333-e2ca015ad7b9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:47.495403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:47.507303 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8646d7b6-843b-4a4a-ab47-a7379bda2956 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:47.507995 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8646d7b6-843b-4a4a-ab47-a7379bda2956 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:47.508500 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8646d7b6-843b-4a4a-ab47-a7379bda2956 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:47.509071 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8646d7b6-843b-4a4a-ab47-a7379bda2956 admin admin] PATH_INFO: `/v3/projects/f7f37ef7959d4674acca179a4797e272` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:47.510397 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8646d7b6-843b-4a4a-ab47-a7379bda2956 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f7f37ef7959d4674acca179a4797e272)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:47.513309 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8646d7b6-843b-4a4a-ab47-a7379bda2956 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:47.556336 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2943746-2b92-4cd7-b333-e2ca015ad7b9 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:47.564283 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-8646d7b6-843b-4a4a-ab47-a7379bda2956 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f7f37ef7959d4674acca179a4797e272', 'request_id': 'req-8646d7b6-843b-4a4a-ab47-a7379bda2956'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:47.571201 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-8646d7b6-843b-4a4a-ab47-a7379bda2956 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f7f37ef7959d4674acca179a4797e272', 'request_id': 'req-8646d7b6-843b-4a4a-ab47-a7379bda2956'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:47.577818 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-8646d7b6-843b-4a4a-ab47-a7379bda2956 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f7f37ef7959d4674acca179a4797e272', 'request_id': 'req-8646d7b6-843b-4a4a-ab47-a7379bda2956'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:47.594463 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-8646d7b6-843b-4a4a-ab47-a7379bda2956 admin admin] The token cache is being invalidate because project f7f37ef7959d4674acca179a4797e272 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:22:47.594888 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-8646d7b6-843b-4a4a-ab47-a7379bda2956 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:47.595681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-8646d7b6-843b-4a4a-ab47-a7379bda2956 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:47.597245 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1186/2396] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:22:47 2022] DELETE /identity/v3/projects/f7f37ef7959d4674acca179a4797e272 => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:22:47.601974 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46fe2cf-b27b-4ee0-b0a8-75b4117cd524 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:47.602268 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46fe2cf-b27b-4ee0-b0a8-75b4117cd524 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:47.602550 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46fe2cf-b27b-4ee0-b0a8-75b4117cd524 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:47.622890 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4385a9c8-13a2-4089-8e10-a153c3a104d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:47.623203 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-a46fe2cf-b27b-4ee0-b0a8-75b4117cd524 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:47.623754 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4385a9c8-13a2-4089-8e10-a153c3a104d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:47.623938 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4385a9c8-13a2-4089-8e10-a153c3a104d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:47.624199 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4385a9c8-13a2-4089-8e10-a153c3a104d4 admin admin] PATH_INFO: `/v3/projects/b073c18a79a34b23a9fbf6149a3dea31` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:47.624407 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-a46fe2cf-b27b-4ee0-b0a8-75b4117cd524 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:47.625349 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4385a9c8-13a2-4089-8e10-a153c3a104d4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b073c18a79a34b23a9fbf6149a3dea31)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:47.627266 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4385a9c8-13a2-4089-8e10-a153c3a104d4 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:47.632793 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1187/2397] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:47 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 16:22:47.641079 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23ea8d1-cbd8-4dda-a9db-ffb1f66c3b03 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:47.641376 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23ea8d1-cbd8-4dda-a9db-ffb1f66c3b03 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:47.641781 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23ea8d1-cbd8-4dda-a9db-ffb1f66c3b03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:47.662481 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-c23ea8d1-cbd8-4dda-a9db-ffb1f66c3b03 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:47.663736 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-c23ea8d1-cbd8-4dda-a9db-ffb1f66c3b03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:47.665903 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:47.682178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4385a9c8-13a2-4089-8e10-a153c3a104d4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b073c18a79a34b23a9fbf6149a3dea31', 'request_id': 'req-4385a9c8-13a2-4089-8e10-a153c3a104d4'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:47.688709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4385a9c8-13a2-4089-8e10-a153c3a104d4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b073c18a79a34b23a9fbf6149a3dea31', 'request_id': 'req-4385a9c8-13a2-4089-8e10-a153c3a104d4'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:47.694467 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4385a9c8-13a2-4089-8e10-a153c3a104d4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b073c18a79a34b23a9fbf6149a3dea31', 'request_id': 'req-4385a9c8-13a2-4089-8e10-a153c3a104d4'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:47.708308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4385a9c8-13a2-4089-8e10-a153c3a104d4 admin admin] The token cache is being invalidate because project b073c18a79a34b23a9fbf6149a3dea31 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:22:47.708604 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4385a9c8-13a2-4089-8e10-a153c3a104d4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:47.709246 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4385a9c8-13a2-4089-8e10-a153c3a104d4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:47.710747 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1211/2398] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:22:47 2022] DELETE /identity/v3/projects/b073c18a79a34b23a9fbf6149a3dea31 => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:22:47.717066 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d73cade-58ba-4cb7-8e2d-961e70d89a6f None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:47.717473 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d73cade-58ba-4cb7-8e2d-961e70d89a6f None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:47.717758 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d73cade-58ba-4cb7-8e2d-961e70d89a6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:47.732452 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1188/2399] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:47 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 16:22:47.736566 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23ea8d1-cbd8-4dda-a9db-ffb1f66c3b03 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:47.738759 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-0d73cade-58ba-4cb7-8e2d-961e70d89a6f None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:47.739997 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-0d73cade-58ba-4cb7-8e2d-961e70d89a6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:47.745583 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23ea8d1-cbd8-4dda-a9db-ffb1f66c3b03 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:47.748223 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1212/2400] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:47 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:22:47.750929 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739eb566-0c42-493b-b0f0-358cfbd16583 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:47.751534 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739eb566-0c42-493b-b0f0-358cfbd16583 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:47.751848 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739eb566-0c42-493b-b0f0-358cfbd16583 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:47.752151 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739eb566-0c42-493b-b0f0-358cfbd16583 admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:47.752806 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739eb566-0c42-493b-b0f0-358cfbd16583 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 16:22:47.753116 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739eb566-0c42-493b-b0f0-358cfbd16583 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:47.754729 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739eb566-0c42-493b-b0f0-358cfbd16583 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:47.756545 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02553bd1-0105-4aa1-9956-8a94dd1574ab None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:47.756922 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02553bd1-0105-4aa1-9956-8a94dd1574ab None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:47.757201 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02553bd1-0105-4aa1-9956-8a94dd1574ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:47.771675 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1189/2401] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 16:22:47 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 16:22:47.772675 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739eb566-0c42-493b-b0f0-358cfbd16583 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:47.777474 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-02553bd1-0105-4aa1-9956-8a94dd1574ab None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:47.779996 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-02553bd1-0105-4aa1-9956-8a94dd1574ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:47.783266 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739eb566-0c42-493b-b0f0-358cfbd16583 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:47.787962 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654ce1b9-7bc6-494e-a1e7-f457612cc80f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:47.788465 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1213/2402] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:47 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 16:22:47.788584 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654ce1b9-7bc6-494e-a1e7-f457612cc80f admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:47.788824 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654ce1b9-7bc6-494e-a1e7-f457612cc80f admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:47.789098 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654ce1b9-7bc6-494e-a1e7-f457612cc80f admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:47.789733 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654ce1b9-7bc6-494e-a1e7-f457612cc80f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:47.791652 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654ce1b9-7bc6-494e-a1e7-f457612cc80f admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:47.792775 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02553bd1-0105-4aa1-9956-8a94dd1574ab None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:47.802482 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02553bd1-0105-4aa1-9956-8a94dd1574ab None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:47.807148 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8067979-7d8d-4c49-8db7-86fa919b809a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:47.807781 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8067979-7d8d-4c49-8db7-86fa919b809a admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:47.808067 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8067979-7d8d-4c49-8db7-86fa919b809a admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:47.808338 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8067979-7d8d-4c49-8db7-86fa919b809a admin admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:47.809061 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8067979-7d8d-4c49-8db7-86fa919b809a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 16:22:47.809578 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8067979-7d8d-4c49-8db7-86fa919b809a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:47.811241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8067979-7d8d-4c49-8db7-86fa919b809a admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:47.822973 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1190/2403] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:22:47 2022] POST /identity/v3/projects => generated 366 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 16:22:47.827882 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654ce1b9-7bc6-494e-a1e7-f457612cc80f admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:47.829722 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1214/2404] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 16:22:47 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 16:22:47.834310 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8067979-7d8d-4c49-8db7-86fa919b809a admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:47.845933 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8067979-7d8d-4c49-8db7-86fa919b809a admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:47.880709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:47.904141 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654ce1b9-7bc6-494e-a1e7-f457612cc80f admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:47.909124 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6609c00-b4a0-457e-b374-5c8c7d86e70b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:47.909673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa1c966-33c3-4cf9-b55c-02e403c2ffb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:47.909806 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6609c00-b4a0-457e-b374-5c8c7d86e70b admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:47.910033 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6609c00-b4a0-457e-b374-5c8c7d86e70b admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:47.910308 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6609c00-b4a0-457e-b374-5c8c7d86e70b admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:47.910415 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa1c966-33c3-4cf9-b55c-02e403c2ffb8 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:47.910763 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa1c966-33c3-4cf9-b55c-02e403c2ffb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:47.910951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6609c00-b4a0-457e-b374-5c8c7d86e70b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:47.911148 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa1c966-33c3-4cf9-b55c-02e403c2ffb8 admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:47.911892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa1c966-33c3-4cf9-b55c-02e403c2ffb8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:47.912986 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6609c00-b4a0-457e-b374-5c8c7d86e70b admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:47.914225 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa1c966-33c3-4cf9-b55c-02e403c2ffb8 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:47.943009 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1191/2405] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:22:47 2022] POST /identity/v3/users => generated 284 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 16:22:47.943876 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6609c00-b4a0-457e-b374-5c8c7d86e70b admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:47.947715 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1215/2406] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:22:47 2022] POST /identity/v3/projects => generated 386 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 16:22:47.947860 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-c6609c00-b4a0-457e-b374-5c8c7d86e70b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:47.948851 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa1c966-33c3-4cf9-b55c-02e403c2ffb8 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:48.005451 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6609c00-b4a0-457e-b374-5c8c7d86e70b admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:48.013710 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa1c966-33c3-4cf9-b55c-02e403c2ffb8 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:48.018830 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1b0fd5-60fa-4e56-b7f0-369a8c930e8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:48.019483 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1b0fd5-60fa-4e56-b7f0-369a8c930e8a admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:48.019817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1b0fd5-60fa-4e56-b7f0-369a8c930e8a admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:48.020098 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1b0fd5-60fa-4e56-b7f0-369a8c930e8a admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:48.020755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1b0fd5-60fa-4e56-b7f0-369a8c930e8a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:48.022443 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1b0fd5-60fa-4e56-b7f0-369a8c930e8a admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:48.037918 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1216/2407] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:22:47 2022] GET /identity/v3/roles => generated 1427 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:22:48.037918 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1b0fd5-60fa-4e56-b7f0-369a8c930e8a admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:48.045889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1b0fd5-60fa-4e56-b7f0-369a8c930e8a admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:48.050595 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939a3270-0ac3-437e-87eb-c71f4c870023 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:48.051192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939a3270-0ac3-437e-87eb-c71f4c870023 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:48.051475 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939a3270-0ac3-437e-87eb-c71f4c870023 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:48.051785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939a3270-0ac3-437e-87eb-c71f4c870023 admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:48.052439 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939a3270-0ac3-437e-87eb-c71f4c870023 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:48.054364 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939a3270-0ac3-437e-87eb-c71f4c870023 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:48.059913 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da683a99-e798-417c-b362-40984c942725 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:48.060652 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da683a99-e798-417c-b362-40984c942725 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:48.060930 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da683a99-e798-417c-b362-40984c942725 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:48.061204 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da683a99-e798-417c-b362-40984c942725 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:48.062999 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-da683a99-e798-417c-b362-40984c942725 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:48.071776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=90,keystone:DELETE=5,keystone:INSERT=5 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:48.077481 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da683a99-e798-417c-b362-40984c942725 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:48.079064 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da683a99-e798-417c-b362-40984c942725 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:48.081070 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1217/2408] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:22:48 2022] POST /identity/v3/users => generated 294 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:22:48.082399 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939a3270-0ac3-437e-87eb-c71f4c870023 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:48.089809 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1192/2409] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:22:47 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:22:48.090796 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da683a99-e798-417c-b362-40984c942725 service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:48.093329 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939a3270-0ac3-437e-87eb-c71f4c870023 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:48.098332 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673f5975-1737-466d-bc07-748b7bad6df5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:48.098972 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673f5975-1737-466d-bc07-748b7bad6df5 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:48.099258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673f5975-1737-466d-bc07-748b7bad6df5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:48.099701 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673f5975-1737-466d-bc07-748b7bad6df5 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:48.100423 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673f5975-1737-466d-bc07-748b7bad6df5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:48.102028 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673f5975-1737-466d-bc07-748b7bad6df5 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:48.102250 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da683a99-e798-417c-b362-40984c942725 service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:48.106928 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890e1628-f4cb-4679-8a9b-f874a0d0ef70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:48.107515 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890e1628-f4cb-4679-8a9b-f874a0d0ef70 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:48.107850 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890e1628-f4cb-4679-8a9b-f874a0d0ef70 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:48.108127 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890e1628-f4cb-4679-8a9b-f874a0d0ef70 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:48.108778 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-890e1628-f4cb-4679-8a9b-f874a0d0ef70 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:48.110360 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-890e1628-f4cb-4679-8a9b-f874a0d0ef70 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:48.117239 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1218/2410] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:22:48 2022] GET /identity/v3/roles => generated 1427 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:22:48.118264 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673f5975-1737-466d-bc07-748b7bad6df5 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:48.126323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1193/2411] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:22:48 2022] GET /identity/v3/roles => generated 1427 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:22:48.128726 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890e1628-f4cb-4679-8a9b-f874a0d0ef70 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:48.137173 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673f5975-1737-466d-bc07-748b7bad6df5 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:48.141723 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890e1628-f4cb-4679-8a9b-f874a0d0ef70 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:48.143442 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8b1812-76ca-4769-a6e2-2cc16f4bfe6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:48.144210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8b1812-76ca-4769-a6e2-2cc16f4bfe6a admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:48.144563 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8b1812-76ca-4769-a6e2-2cc16f4bfe6a admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:48.144920 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8b1812-76ca-4769-a6e2-2cc16f4bfe6a admin admin] PATH_INFO: `/v3/projects/5d0f97e2d1654e9a899c645151123520/users/93b8d0c807664c4488400401efcee801/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:48.147314 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4eb8319-15c5-4d58-bc3b-0719f8d1d860 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:48.147928 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4eb8319-15c5-4d58-bc3b-0719f8d1d860 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:48.149080 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4eb8319-15c5-4d58-bc3b-0719f8d1d860 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:48.149080 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4eb8319-15c5-4d58-bc3b-0719f8d1d860 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:48.149339 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4eb8319-15c5-4d58-bc3b-0719f8d1d860 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:48.151040 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4eb8319-15c5-4d58-bc3b-0719f8d1d860 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:48.165231 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1194/2412] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:22:48 2022] GET /identity/v3/roles => generated 1427 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 16:22:48.165445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8b1812-76ca-4769-a6e2-2cc16f4bfe6a admin admin] RBAC: Authorizing `identity:create_grant(project_id=5d0f97e2d1654e9a899c645151123520, user_id=93b8d0c807664c4488400401efcee801, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:48.165445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8b1812-76ca-4769-a6e2-2cc16f4bfe6a admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:48.166536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4eb8319-15c5-4d58-bc3b-0719f8d1d860 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:48.181467 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4eb8319-15c5-4d58-bc3b-0719f8d1d860 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:48.194151 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1219/2413] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:22:48 2022] PUT /identity/v3/projects/5d0f97e2d1654e9a899c645151123520/users/93b8d0c807664c4488400401efcee801/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:22:48.196335 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e831aed-eab0-4b3b-b655-585b64018d5b None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:48.196745 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e831aed-eab0-4b3b-b655-585b64018d5b None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:48.200035 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e831aed-eab0-4b3b-b655-585b64018d5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:48.240968 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-7e831aed-eab0-4b3b-b655-585b64018d5b None None] MFA Rules not processed for user `93b8d0c807664c4488400401efcee801`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:48.242206 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-7e831aed-eab0-4b3b-b655-585b64018d5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:48.257978 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44922322-f371-4b7e-9b29-6ee019d73bfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:48.258569 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44922322-f371-4b7e-9b29-6ee019d73bfa admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:48.258852 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44922322-f371-4b7e-9b29-6ee019d73bfa admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:48.259130 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44922322-f371-4b7e-9b29-6ee019d73bfa admin admin] PATH_INFO: `/v3/projects/d34bbff272dd417894d0dd411b81aa13/users/b0752dfc9bdd4ab3af8859e38001ded3/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:48.279596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44922322-f371-4b7e-9b29-6ee019d73bfa admin admin] RBAC: Authorizing `identity:create_grant(project_id=d34bbff272dd417894d0dd411b81aa13, user_id=b0752dfc9bdd4ab3af8859e38001ded3, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:48.285868 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44922322-f371-4b7e-9b29-6ee019d73bfa admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:48.288661 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:48.305208 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1195/2414] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:22:48 2022] PUT /identity/v3/projects/d34bbff272dd417894d0dd411b81aa13/users/b0752dfc9bdd4ab3af8859e38001ded3/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:22:48.316617 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb4a2e4-882f-40a6-8f49-241babf363e7 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:48.318640 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb4a2e4-882f-40a6-8f49-241babf363e7 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:48.320001 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb4a2e4-882f-40a6-8f49-241babf363e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:48.334437 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1220/2415] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:48 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 16:22:48.339322 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e831aed-eab0-4b3b-b655-585b64018d5b None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:48.350518 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e831aed-eab0-4b3b-b655-585b64018d5b None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:48.353712 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-ebb4a2e4-882f-40a6-8f49-241babf363e7 None None] MFA Rules not processed for user `b0752dfc9bdd4ab3af8859e38001ded3`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:48.355312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-ebb4a2e4-882f-40a6-8f49-241babf363e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:48.416931 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d9b931-d5e2-4f6d-808a-ae029b3c4798 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:48.417639 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d9b931-d5e2-4f6d-808a-ae029b3c4798 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:48.418016 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d9b931-d5e2-4f6d-808a-ae029b3c4798 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:48.418292 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d9b931-d5e2-4f6d-808a-ae029b3c4798 admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:48.419707 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d9b931-d5e2-4f6d-808a-ae029b3c4798 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:48.422032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d9b931-d5e2-4f6d-808a-ae029b3c4798 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:48.429258 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1196/2416] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:48 2022] POST /identity/v3/auth/tokens => generated 3272 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 16:22:48.435120 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb4a2e4-882f-40a6-8f49-241babf363e7 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:48.449850 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb4a2e4-882f-40a6-8f49-241babf363e7 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:48.455459 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0168c60d-0080-4f5b-a0d4-75c51adfcee8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:48.457789 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0168c60d-0080-4f5b-a0d4-75c51adfcee8 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:48.457789 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0168c60d-0080-4f5b-a0d4-75c51adfcee8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:48.457789 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0168c60d-0080-4f5b-a0d4-75c51adfcee8 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:48.457789 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0168c60d-0080-4f5b-a0d4-75c51adfcee8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:48.459765 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0168c60d-0080-4f5b-a0d4-75c51adfcee8 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:48.463231 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1221/2417] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:22:48 2022] POST /identity/v3/projects => generated 366 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 16:22:48.465219 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d9b931-d5e2-4f6d-808a-ae029b3c4798 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:48.496096 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1197/2418] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:22:48 2022] POST /identity/v3/projects => generated 386 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 16:22:48.510191 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0168c60d-0080-4f5b-a0d4-75c51adfcee8 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:48.523246 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:48.563469 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d9b931-d5e2-4f6d-808a-ae029b3c4798 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:48.588757 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0168c60d-0080-4f5b-a0d4-75c51adfcee8 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:48.595275 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0775e5-8161-4d2b-8369-734042db9965 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:48.596051 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0775e5-8161-4d2b-8369-734042db9965 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:48.596333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0775e5-8161-4d2b-8369-734042db9965 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:48.596610 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0775e5-8161-4d2b-8369-734042db9965 admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:48.597738 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0775e5-8161-4d2b-8369-734042db9965 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:48.601366 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0775e5-8161-4d2b-8369-734042db9965 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:48.629862 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1198/2419] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:22:48 2022] POST /identity/v3/users => generated 300 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 16:22:48.633684 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0775e5-8161-4d2b-8369-734042db9965 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:48.641860 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2609df0-3431-4511-a9cb-11cb4e903669 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:48.642475 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2609df0-3431-4511-a9cb-11cb4e903669 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:48.642957 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2609df0-3431-4511-a9cb-11cb4e903669 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:48.643078 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2609df0-3431-4511-a9cb-11cb4e903669 admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:48.643850 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2609df0-3431-4511-a9cb-11cb4e903669 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:48.645837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2609df0-3431-4511-a9cb-11cb4e903669 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:48.646207 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0775e5-8161-4d2b-8369-734042db9965 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:48.652355 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40a8a77-f556-4191-a956-de7d604e8d1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:48.653634 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40a8a77-f556-4191-a956-de7d604e8d1a admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:48.653985 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40a8a77-f556-4191-a956-de7d604e8d1a admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:48.654524 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40a8a77-f556-4191-a956-de7d604e8d1a admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:48.655278 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f40a8a77-f556-4191-a956-de7d604e8d1a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:48.657497 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f40a8a77-f556-4191-a956-de7d604e8d1a admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:48.670694 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1199/2420] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:22:48 2022] GET /identity/v3/roles => generated 1427 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:22:48.674603 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40a8a77-f556-4191-a956-de7d604e8d1a admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:48.676699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1222/2421] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:22:48 2022] POST /identity/v3/users => generated 290 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:22:48.677689 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2609df0-3431-4511-a9cb-11cb4e903669 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:48.686297 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40a8a77-f556-4191-a956-de7d604e8d1a admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:48.689408 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2609df0-3431-4511-a9cb-11cb4e903669 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:48.691199 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd58232-7230-43dd-b347-9e94fcd390f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:48.691891 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd58232-7230-43dd-b347-9e94fcd390f2 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:48.692176 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd58232-7230-43dd-b347-9e94fcd390f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:48.692565 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd58232-7230-43dd-b347-9e94fcd390f2 admin admin] PATH_INFO: `/v3/projects/ddc40de9a12b4254a05de16b6955f6b5/users/0dfaaec302cc4b1b900c61b13a1b964e/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:48.695237 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d787495-b242-4901-8602-753b41f0a07c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:48.696773 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d787495-b242-4901-8602-753b41f0a07c admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:48.697070 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d787495-b242-4901-8602-753b41f0a07c admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:48.697936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d787495-b242-4901-8602-753b41f0a07c admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:48.698613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d787495-b242-4901-8602-753b41f0a07c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:48.700858 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d787495-b242-4901-8602-753b41f0a07c admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:48.710496 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd58232-7230-43dd-b347-9e94fcd390f2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ddc40de9a12b4254a05de16b6955f6b5, user_id=0dfaaec302cc4b1b900c61b13a1b964e, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:48.711211 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1223/2422] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:22:48 2022] GET /identity/v3/roles => generated 1427 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:22:48.714669 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd58232-7230-43dd-b347-9e94fcd390f2 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:48.716221 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d787495-b242-4901-8602-753b41f0a07c admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:48.729018 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d787495-b242-4901-8602-753b41f0a07c admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:48.729835 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1200/2423] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:22:48 2022] PUT /identity/v3/projects/ddc40de9a12b4254a05de16b6955f6b5/users/0dfaaec302cc4b1b900c61b13a1b964e/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:22:48.734302 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd58232-7230-43dd-b347-9e94fcd390f2 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:48.758303 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:INSERT=6,keystone:SELECT=94 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:48.800240 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d22f2d9-33fe-40a3-81b3-235c4e74dc1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:48.801026 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d22f2d9-33fe-40a3-81b3-235c4e74dc1f admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:48.801315 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d22f2d9-33fe-40a3-81b3-235c4e74dc1f admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:48.801596 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d22f2d9-33fe-40a3-81b3-235c4e74dc1f admin admin] PATH_INFO: `/v3/projects/9efc7f1cd12648ad87b86764887bde67/users/feffac2c641d44f08156cc053039c7c8/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:48.816489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd58232-7230-43dd-b347-9e94fcd390f2 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:48.819662 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d22f2d9-33fe-40a3-81b3-235c4e74dc1f admin admin] RBAC: Authorizing `identity:create_grant(project_id=9efc7f1cd12648ad87b86764887bde67, user_id=feffac2c641d44f08156cc053039c7c8, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:48.822060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6a0a10-b6f2-4c93-b2c6-4369a58de71f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:48.822670 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6a0a10-b6f2-4c93-b2c6-4369a58de71f admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:48.822946 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6a0a10-b6f2-4c93-b2c6-4369a58de71f admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:48.823216 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6a0a10-b6f2-4c93-b2c6-4369a58de71f admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:48.823870 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d22f2d9-33fe-40a3-81b3-235c4e74dc1f admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:48.823992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6a0a10-b6f2-4c93-b2c6-4369a58de71f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:48.826371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6a0a10-b6f2-4c93-b2c6-4369a58de71f admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:48.838356 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1201/2424] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:22:48 2022] GET /identity/v3/roles => generated 1427 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 16:22:48.843389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1224/2425] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:22:48 2022] PUT /identity/v3/projects/9efc7f1cd12648ad87b86764887bde67/users/feffac2c641d44f08156cc053039c7c8/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:22:48.846254 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6a0a10-b6f2-4c93-b2c6-4369a58de71f admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:48.847169 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d22f2d9-33fe-40a3-81b3-235c4e74dc1f admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:48.857481 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6a0a10-b6f2-4c93-b2c6-4369a58de71f admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:48.907476 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:48.926934 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d22f2d9-33fe-40a3-81b3-235c4e74dc1f admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:48.931536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9261f340-969a-4030-b4cc-4ed209135846 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:48.932297 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9261f340-969a-4030-b4cc-4ed209135846 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:48.932577 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9261f340-969a-4030-b4cc-4ed209135846 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:48.932848 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9261f340-969a-4030-b4cc-4ed209135846 admin admin] PATH_INFO: `/v3/projects/ddc40de9a12b4254a05de16b6955f6b5/users/0dfaaec302cc4b1b900c61b13a1b964e/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:48.935338 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9261f340-969a-4030-b4cc-4ed209135846 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ddc40de9a12b4254a05de16b6955f6b5, user_id=0dfaaec302cc4b1b900c61b13a1b964e, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:48.937038 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc0e935-3bec-4473-9f6d-189f6305ea4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:48.938011 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc0e935-3bec-4473-9f6d-189f6305ea4a admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:48.938120 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9261f340-969a-4030-b4cc-4ed209135846 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:48.938507 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc0e935-3bec-4473-9f6d-189f6305ea4a admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:48.938989 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc0e935-3bec-4473-9f6d-189f6305ea4a admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:48.939758 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc0e935-3bec-4473-9f6d-189f6305ea4a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:48.941740 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc0e935-3bec-4473-9f6d-189f6305ea4a admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:48.948795 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1202/2426] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:22:48 2022] PUT /identity/v3/projects/ddc40de9a12b4254a05de16b6955f6b5/users/0dfaaec302cc4b1b900c61b13a1b964e/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:22:48.958150 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1225/2427] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:22:48 2022] GET /identity/v3/roles => generated 1427 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:22:48.960166 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c689440-9d16-47a2-89e4-d9ed6e178d2f None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:48.960301 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc0e935-3bec-4473-9f6d-189f6305ea4a admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:48.960459 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c689440-9d16-47a2-89e4-d9ed6e178d2f None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:48.960730 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c689440-9d16-47a2-89e4-d9ed6e178d2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:48.973405 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc0e935-3bec-4473-9f6d-189f6305ea4a admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:49.046870 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee6f9ee-aa45-4fb4-a6b8-11234ea19487 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:49.047677 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee6f9ee-aa45-4fb4-a6b8-11234ea19487 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:49.048031 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee6f9ee-aa45-4fb4-a6b8-11234ea19487 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:49.048371 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee6f9ee-aa45-4fb4-a6b8-11234ea19487 admin admin] PATH_INFO: `/v3/projects/9efc7f1cd12648ad87b86764887bde67/users/feffac2c641d44f08156cc053039c7c8/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:49.050593 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee6f9ee-aa45-4fb4-a6b8-11234ea19487 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9efc7f1cd12648ad87b86764887bde67, user_id=feffac2c641d44f08156cc053039c7c8, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:49.053387 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee6f9ee-aa45-4fb4-a6b8-11234ea19487 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:49.066775 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1226/2428] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:22:48 2022] PUT /identity/v3/projects/9efc7f1cd12648ad87b86764887bde67/users/feffac2c641d44f08156cc053039c7c8/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:22:49.077267 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbd725f-759f-40cf-b328-ea77b1237def None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:49.077566 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbd725f-759f-40cf-b328-ea77b1237def None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:49.077979 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbd725f-759f-40cf-b328-ea77b1237def None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:49.109984 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-4dbd725f-759f-40cf-b328-ea77b1237def None None] MFA Rules not processed for user `feffac2c641d44f08156cc053039c7c8`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:49.111327 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-4dbd725f-759f-40cf-b328-ea77b1237def None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:49.113436 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-5c689440-9d16-47a2-89e4-d9ed6e178d2f None None] MFA Rules not processed for user `0dfaaec302cc4b1b900c61b13a1b964e`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:49.114610 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-5c689440-9d16-47a2-89e4-d9ed6e178d2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:49.189321 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1227/2429] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:49 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 16:22:49.195683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1203/2430] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:48 2022] POST /identity/v3/auth/tokens => generated 3339 bytes in 236 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 16:22:49.204407 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dbd725f-759f-40cf-b328-ea77b1237def None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:49.205850 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-4dbd725f-759f-40cf-b328-ea77b1237def None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:49.208116 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c689440-9d16-47a2-89e4-d9ed6e178d2f None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:49.209743 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-5c689440-9d16-47a2-89e4-d9ed6e178d2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:49.249233 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:49.288735 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dbd725f-759f-40cf-b328-ea77b1237def None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:49.292869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c689440-9d16-47a2-89e4-d9ed6e178d2f None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:49.293039 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84e3884-1b50-4634-aba9-5119bd951080 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:49.293517 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84e3884-1b50-4634-aba9-5119bd951080 service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:49.294308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84e3884-1b50-4634-aba9-5119bd951080 service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:49.294308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84e3884-1b50-4634-aba9-5119bd951080 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:49.298244 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93e07d0-f34c-4c43-90cb-ac976ba19ed3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:49.298881 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93e07d0-f34c-4c43-90cb-ac976ba19ed3 service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:49.299179 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93e07d0-f34c-4c43-90cb-ac976ba19ed3 service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:49.299451 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93e07d0-f34c-4c43-90cb-ac976ba19ed3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:49.302087 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:49.306726 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84e3884-1b50-4634-aba9-5119bd951080 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:49.309430 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84e3884-1b50-4634-aba9-5119bd951080 service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:49.310212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b93e07d0-f34c-4c43-90cb-ac976ba19ed3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:49.311744 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b93e07d0-f34c-4c43-90cb-ac976ba19ed3 service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:49.324292 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1228/2431] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:22:49 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:22:49.341384 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1204/2432] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:22:49 2022] GET /identity/v3/auth/tokens => generated 3339 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:22:49.424749 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84e3884-1b50-4634-aba9-5119bd951080 service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:49.435522 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84e3884-1b50-4634-aba9-5119bd951080 service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:49.439498 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4fb3c4-ffe9-4382-a1a3-caefe4bab438 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:49.440909 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4fb3c4-ffe9-4382-a1a3-caefe4bab438 service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:49.441263 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4fb3c4-ffe9-4382-a1a3-caefe4bab438 service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:49.441609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4fb3c4-ffe9-4382-a1a3-caefe4bab438 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:49.519697 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4fb3c4-ffe9-4382-a1a3-caefe4bab438 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:49.521251 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4fb3c4-ffe9-4382-a1a3-caefe4bab438 service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:49.531594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1229/2433] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:22:49 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:22:50.098990 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93e07d0-f34c-4c43-90cb-ac976ba19ed3 service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:50.183537 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93e07d0-f34c-4c43-90cb-ac976ba19ed3 service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:50.188842 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083cc651-d7cb-498a-9701-bd91249e2a6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:50.189810 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083cc651-d7cb-498a-9701-bd91249e2a6d admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:50.189810 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083cc651-d7cb-498a-9701-bd91249e2a6d admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:50.190131 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083cc651-d7cb-498a-9701-bd91249e2a6d admin admin] PATH_INFO: `/v3/users/b0752dfc9bdd4ab3af8859e38001ded3` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:50.191783 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083cc651-d7cb-498a-9701-bd91249e2a6d admin admin] RBAC: Authorizing `identity:delete_user(user_id=b0752dfc9bdd4ab3af8859e38001ded3)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:50.193912 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083cc651-d7cb-498a-9701-bd91249e2a6d admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:50.260564 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-083cc651-d7cb-498a-9701-bd91249e2a6d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b0752dfc9bdd4ab3af8859e38001ded3'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:50.261594 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-083cc651-d7cb-498a-9701-bd91249e2a6d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b0752dfc9bdd4ab3af8859e38001ded3'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:50.271458 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-083cc651-d7cb-498a-9701-bd91249e2a6d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b0752dfc9bdd4ab3af8859e38001ded3'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:50.289486 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-083cc651-d7cb-498a-9701-bd91249e2a6d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b0752dfc9bdd4ab3af8859e38001ded3'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:50.290851 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-083cc651-d7cb-498a-9701-bd91249e2a6d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b0752dfc9bdd4ab3af8859e38001ded3'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:50.308256 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1205/2434] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:22:50 2022] DELETE /identity/v3/users/b0752dfc9bdd4ab3af8859e38001ded3 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:22:50.311980 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8354d1-97a1-4552-9c6e-bb0a4c6f7dc9 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:50.312489 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8354d1-97a1-4552-9c6e-bb0a4c6f7dc9 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:50.312588 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8354d1-97a1-4552-9c6e-bb0a4c6f7dc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:50.336763 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-ad8354d1-97a1-4552-9c6e-bb0a4c6f7dc9 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:50.338116 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-ad8354d1-97a1-4552-9c6e-bb0a4c6f7dc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:50.407673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1230/2435] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:50 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:22:50.417709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083cc651-d7cb-498a-9701-bd91249e2a6d admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:50.420207 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-083cc651-d7cb-498a-9701-bd91249e2a6d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:50.477517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:50.497965 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083cc651-d7cb-498a-9701-bd91249e2a6d admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:50.502405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1cfffc8-2ece-4abc-b1ff-15b5b83a8ae1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:50.503010 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cfffc8-2ece-4abc-b1ff-15b5b83a8ae1 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:50.503288 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cfffc8-2ece-4abc-b1ff-15b5b83a8ae1 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:50.503686 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cfffc8-2ece-4abc-b1ff-15b5b83a8ae1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:50.512939 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1cfffc8-2ece-4abc-b1ff-15b5b83a8ae1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:50.514402 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1cfffc8-2ece-4abc-b1ff-15b5b83a8ae1 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:50.525061 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1206/2436] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:22:50 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 16:22:51.084545 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8354d1-97a1-4552-9c6e-bb0a4c6f7dc9 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:51.086166 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-ad8354d1-97a1-4552-9c6e-bb0a4c6f7dc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:51.110317 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8354d1-97a1-4552-9c6e-bb0a4c6f7dc9 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:51.115980 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b31e32d-9c95-48dd-a9c7-54cb6d567322 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:51.116656 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b31e32d-9c95-48dd-a9c7-54cb6d567322 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:51.117010 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b31e32d-9c95-48dd-a9c7-54cb6d567322 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:51.117351 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b31e32d-9c95-48dd-a9c7-54cb6d567322 admin admin] PATH_INFO: `/v3/projects/d34bbff272dd417894d0dd411b81aa13` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:51.118596 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b31e32d-9c95-48dd-a9c7-54cb6d567322 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d34bbff272dd417894d0dd411b81aa13)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:51.120646 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b31e32d-9c95-48dd-a9c7-54cb6d567322 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:51.177315 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-3b31e32d-9c95-48dd-a9c7-54cb6d567322 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd34bbff272dd417894d0dd411b81aa13', 'request_id': 'req-3b31e32d-9c95-48dd-a9c7-54cb6d567322'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:51.193468 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-3b31e32d-9c95-48dd-a9c7-54cb6d567322 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd34bbff272dd417894d0dd411b81aa13', 'request_id': 'req-3b31e32d-9c95-48dd-a9c7-54cb6d567322'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:51.204300 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-3b31e32d-9c95-48dd-a9c7-54cb6d567322 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd34bbff272dd417894d0dd411b81aa13', 'request_id': 'req-3b31e32d-9c95-48dd-a9c7-54cb6d567322'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:51.233311 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-3b31e32d-9c95-48dd-a9c7-54cb6d567322 admin admin] The token cache is being invalidate because project d34bbff272dd417894d0dd411b81aa13 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:22:51.233944 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-3b31e32d-9c95-48dd-a9c7-54cb6d567322 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:51.235006 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-3b31e32d-9c95-48dd-a9c7-54cb6d567322 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:51.239182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1231/2437] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:22:51 2022] DELETE /identity/v3/projects/d34bbff272dd417894d0dd411b81aa13 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:22:51.244407 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1cfffc8-2ece-4abc-b1ff-15b5b83a8ae1 service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:51.255307 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1cfffc8-2ece-4abc-b1ff-15b5b83a8ae1 service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:51.362965 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736ab583-6068-46cc-a3ae-6ff98b9ae436 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:51.363811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736ab583-6068-46cc-a3ae-6ff98b9ae436 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:51.364254 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736ab583-6068-46cc-a3ae-6ff98b9ae436 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:51.364536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736ab583-6068-46cc-a3ae-6ff98b9ae436 admin admin] PATH_INFO: `/v3/users/0dfaaec302cc4b1b900c61b13a1b964e` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:51.365701 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736ab583-6068-46cc-a3ae-6ff98b9ae436 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0dfaaec302cc4b1b900c61b13a1b964e)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:51.368325 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736ab583-6068-46cc-a3ae-6ff98b9ae436 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:51.435400 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-736ab583-6068-46cc-a3ae-6ff98b9ae436 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0dfaaec302cc4b1b900c61b13a1b964e'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:51.436308 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-736ab583-6068-46cc-a3ae-6ff98b9ae436 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0dfaaec302cc4b1b900c61b13a1b964e'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:51.445257 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-736ab583-6068-46cc-a3ae-6ff98b9ae436 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0dfaaec302cc4b1b900c61b13a1b964e'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:51.469180 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-736ab583-6068-46cc-a3ae-6ff98b9ae436 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0dfaaec302cc4b1b900c61b13a1b964e'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:51.470346 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-736ab583-6068-46cc-a3ae-6ff98b9ae436 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0dfaaec302cc4b1b900c61b13a1b964e'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:51.486519 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1207/2438] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:22:51 2022] DELETE /identity/v3/users/0dfaaec302cc4b1b900c61b13a1b964e => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:22:51.857490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b31e32d-9c95-48dd-a9c7-54cb6d567322 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:51.859164 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-3b31e32d-9c95-48dd-a9c7-54cb6d567322 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:51.881998 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b31e32d-9c95-48dd-a9c7-54cb6d567322 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:51.897895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:51.911494 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736ab583-6068-46cc-a3ae-6ff98b9ae436 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:51.913129 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-736ab583-6068-46cc-a3ae-6ff98b9ae436 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:51.955462 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5c79bd-7816-404f-bc63-d1642aca89af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:51.956079 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5c79bd-7816-404f-bc63-d1642aca89af admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:51.956367 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5c79bd-7816-404f-bc63-d1642aca89af admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:51.956646 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5c79bd-7816-404f-bc63-d1642aca89af admin admin] PATH_INFO: `/v3/projects/ddc40de9a12b4254a05de16b6955f6b5` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:51.957796 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea5c79bd-7816-404f-bc63-d1642aca89af admin admin] RBAC: Authorizing `identity:delete_project(project_id=ddc40de9a12b4254a05de16b6955f6b5)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:51.959832 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea5c79bd-7816-404f-bc63-d1642aca89af admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:51.987397 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736ab583-6068-46cc-a3ae-6ff98b9ae436 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:51.993117 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7332ef-c634-45a3-a844-4043ef38877e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:51.993858 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7332ef-c634-45a3-a844-4043ef38877e admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:51.994213 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7332ef-c634-45a3-a844-4043ef38877e admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:51.994556 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7332ef-c634-45a3-a844-4043ef38877e admin admin] PATH_INFO: `/v3/users/2fe5724620da46ddbefbdfd442dc8d73` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:51.995710 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a7332ef-c634-45a3-a844-4043ef38877e admin admin] RBAC: Authorizing `identity:delete_user(user_id=2fe5724620da46ddbefbdfd442dc8d73)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:51.998485 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a7332ef-c634-45a3-a844-4043ef38877e admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:52.008714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:52.011424 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-ea5c79bd-7816-404f-bc63-d1642aca89af admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ddc40de9a12b4254a05de16b6955f6b5', 'request_id': 'req-ea5c79bd-7816-404f-bc63-d1642aca89af'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:52.018056 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-ea5c79bd-7816-404f-bc63-d1642aca89af admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ddc40de9a12b4254a05de16b6955f6b5', 'request_id': 'req-ea5c79bd-7816-404f-bc63-d1642aca89af'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:52.024642 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-ea5c79bd-7816-404f-bc63-d1642aca89af admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ddc40de9a12b4254a05de16b6955f6b5', 'request_id': 'req-ea5c79bd-7816-404f-bc63-d1642aca89af'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:52.038652 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-ea5c79bd-7816-404f-bc63-d1642aca89af admin admin] The token cache is being invalidate because project ddc40de9a12b4254a05de16b6955f6b5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:22:52.039014 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-ea5c79bd-7816-404f-bc63-d1642aca89af admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:52.039778 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-ea5c79bd-7816-404f-bc63-d1642aca89af admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:52.041507 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1232/2439] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:22:51 2022] DELETE /identity/v3/projects/ddc40de9a12b4254a05de16b6955f6b5 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:22:52.047077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652381f6-0d1a-47e1-91fe-31e7d203f650 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:52.047077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652381f6-0d1a-47e1-91fe-31e7d203f650 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:52.047530 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652381f6-0d1a-47e1-91fe-31e7d203f650 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:52.055042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-2a7332ef-c634-45a3-a844-4043ef38877e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2fe5724620da46ddbefbdfd442dc8d73'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:52.055736 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-2a7332ef-c634-45a3-a844-4043ef38877e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2fe5724620da46ddbefbdfd442dc8d73'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:52.064256 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-2a7332ef-c634-45a3-a844-4043ef38877e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2fe5724620da46ddbefbdfd442dc8d73'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:52.067917 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-652381f6-0d1a-47e1-91fe-31e7d203f650 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:52.069136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-652381f6-0d1a-47e1-91fe-31e7d203f650 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:52.076878 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-2a7332ef-c634-45a3-a844-4043ef38877e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2fe5724620da46ddbefbdfd442dc8d73'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:52.077685 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-2a7332ef-c634-45a3-a844-4043ef38877e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2fe5724620da46ddbefbdfd442dc8d73'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:52.092475 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1208/2440] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:22:51 2022] DELETE /identity/v3/users/2fe5724620da46ddbefbdfd442dc8d73 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:22:52.130851 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1233/2441] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:52 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 16:22:52.198996 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3587faf-31e9-4b0c-9759-eae7064ef4fc None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:52.199554 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3587faf-31e9-4b0c-9759-eae7064ef4fc None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:52.200648 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3587faf-31e9-4b0c-9759-eae7064ef4fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:52.224412 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-f3587faf-31e9-4b0c-9759-eae7064ef4fc None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:52.226121 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-f3587faf-31e9-4b0c-9759-eae7064ef4fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:52.290204 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652381f6-0d1a-47e1-91fe-31e7d203f650 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:52.291772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-652381f6-0d1a-47e1-91fe-31e7d203f650 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:52.317921 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1209/2442] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:52 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:22:52.322125 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3587faf-31e9-4b0c-9759-eae7064ef4fc None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:52.324960 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652381f6-0d1a-47e1-91fe-31e7d203f650 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:52.331426 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90dd64f7-5a50-4115-8338-403a580b8a4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:52.332509 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3587faf-31e9-4b0c-9759-eae7064ef4fc None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:52.338287 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c3a84c-f2b3-4ff2-8950-9a949ba29dbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:52.339898 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c3a84c-f2b3-4ff2-8950-9a949ba29dbb admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:52.340822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c3a84c-f2b3-4ff2-8950-9a949ba29dbb admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:52.341194 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c3a84c-f2b3-4ff2-8950-9a949ba29dbb admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:52.342057 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c3a84c-f2b3-4ff2-8950-9a949ba29dbb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 16:22:52.342401 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90dd64f7-5a50-4115-8338-403a580b8a4d admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:52.342524 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c3a84c-f2b3-4ff2-8950-9a949ba29dbb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:52.342688 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90dd64f7-5a50-4115-8338-403a580b8a4d admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:52.342964 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90dd64f7-5a50-4115-8338-403a580b8a4d admin admin] PATH_INFO: `/v3/projects/8fe57b34d5144e91bafb66f54169cbcf` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:52.344680 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c3a84c-f2b3-4ff2-8950-9a949ba29dbb admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:52.362813 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90dd64f7-5a50-4115-8338-403a580b8a4d admin admin] RBAC: Authorizing `identity:delete_project(project_id=8fe57b34d5144e91bafb66f54169cbcf)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:52.363241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1210/2443] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 16:22:52 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 16:22:52.364231 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c3a84c-f2b3-4ff2-8950-9a949ba29dbb admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:52.366051 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90dd64f7-5a50-4115-8338-403a580b8a4d admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:52.391174 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c3a84c-f2b3-4ff2-8950-9a949ba29dbb admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:52.391174 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9669328b-5389-4d2a-bc26-a32a979aa060 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:52.391174 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9669328b-5389-4d2a-bc26-a32a979aa060 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:52.391174 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9669328b-5389-4d2a-bc26-a32a979aa060 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:52.391787 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9669328b-5389-4d2a-bc26-a32a979aa060 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:52.391787 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9669328b-5389-4d2a-bc26-a32a979aa060 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:52.391787 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9669328b-5389-4d2a-bc26-a32a979aa060 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:52.425591 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:52.426258 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1211/2444] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:22:52 2022] POST /identity/v3/projects => generated 382 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:22:52.427339 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9669328b-5389-4d2a-bc26-a32a979aa060 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:52.440923 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-90dd64f7-5a50-4115-8338-403a580b8a4d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8fe57b34d5144e91bafb66f54169cbcf', 'request_id': 'req-90dd64f7-5a50-4115-8338-403a580b8a4d'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:52.452772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-90dd64f7-5a50-4115-8338-403a580b8a4d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8fe57b34d5144e91bafb66f54169cbcf', 'request_id': 'req-90dd64f7-5a50-4115-8338-403a580b8a4d'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:52.461084 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-90dd64f7-5a50-4115-8338-403a580b8a4d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8fe57b34d5144e91bafb66f54169cbcf', 'request_id': 'req-90dd64f7-5a50-4115-8338-403a580b8a4d'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:52.478979 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-90dd64f7-5a50-4115-8338-403a580b8a4d admin admin] The token cache is being invalidate because project 8fe57b34d5144e91bafb66f54169cbcf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:22:52.479421 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-90dd64f7-5a50-4115-8338-403a580b8a4d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:52.480161 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-90dd64f7-5a50-4115-8338-403a580b8a4d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:52.482706 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1234/2445] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:22:52 2022] DELETE /identity/v3/projects/8fe57b34d5144e91bafb66f54169cbcf => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:22:52.486023 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:52.488946 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748e90f0-17b0-446f-8fe7-85ee1c069c53 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:52.489232 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748e90f0-17b0-446f-8fe7-85ee1c069c53 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:52.489671 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748e90f0-17b0-446f-8fe7-85ee1c069c53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:52.506476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9669328b-5389-4d2a-bc26-a32a979aa060 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:52.511274 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27e7210-8322-4731-8e8b-3e4b7322099b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:52.512130 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27e7210-8322-4731-8e8b-3e4b7322099b admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:52.512275 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27e7210-8322-4731-8e8b-3e4b7322099b admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:52.512554 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27e7210-8322-4731-8e8b-3e4b7322099b admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:52.512652 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-748e90f0-17b0-446f-8fe7-85ee1c069c53 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:52.513321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27e7210-8322-4731-8e8b-3e4b7322099b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:52.514700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-748e90f0-17b0-446f-8fe7-85ee1c069c53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:52.515651 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27e7210-8322-4731-8e8b-3e4b7322099b admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:52.523188 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1235/2446] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:52 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 16:22:52.532816 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ef495f-a958-4506-83a3-3a92d1eaf63f None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:52.533111 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ef495f-a958-4506-83a3-3a92d1eaf63f None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:52.533389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ef495f-a958-4506-83a3-3a92d1eaf63f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:52.539590 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1212/2447] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:22:52 2022] POST /identity/v3/users => generated 292 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:22:52.544401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27e7210-8322-4731-8e8b-3e4b7322099b admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:52.546178 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-a27e7210-8322-4731-8e8b-3e4b7322099b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:52.558164 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-30ef495f-a958-4506-83a3-3a92d1eaf63f None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:52.560651 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-30ef495f-a958-4506-83a3-3a92d1eaf63f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:52.563328 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27e7210-8322-4731-8e8b-3e4b7322099b admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:52.568917 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3457bb0-db1e-4a33-800f-d45c440bfa5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:52.569753 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3457bb0-db1e-4a33-800f-d45c440bfa5b admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:52.570181 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3457bb0-db1e-4a33-800f-d45c440bfa5b admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:52.570461 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3457bb0-db1e-4a33-800f-d45c440bfa5b admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:52.571117 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3457bb0-db1e-4a33-800f-d45c440bfa5b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:52.572129 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1236/2448] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:52 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 16:22:52.572809 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3457bb0-db1e-4a33-800f-d45c440bfa5b admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:52.577836 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ef495f-a958-4506-83a3-3a92d1eaf63f None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:52.584556 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1213/2449] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:22:52 2022] GET /identity/v3/roles => generated 1427 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 16:22:52.588066 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ef495f-a958-4506-83a3-3a92d1eaf63f None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:52.588616 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3457bb0-db1e-4a33-800f-d45c440bfa5b admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:52.597224 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7eacf82-8129-41d3-abde-47abf19a2ab4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:52.597953 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7eacf82-8129-41d3-abde-47abf19a2ab4 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:52.598366 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7eacf82-8129-41d3-abde-47abf19a2ab4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:52.598657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7eacf82-8129-41d3-abde-47abf19a2ab4 admin admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:52.599326 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7eacf82-8129-41d3-abde-47abf19a2ab4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 16:22:52.599666 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7eacf82-8129-41d3-abde-47abf19a2ab4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:52.600191 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3457bb0-db1e-4a33-800f-d45c440bfa5b admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:52.601392 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7eacf82-8129-41d3-abde-47abf19a2ab4 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:52.606631 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be706e4a-7cc1-4bbf-9123-148a4bfccf43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:52.607310 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be706e4a-7cc1-4bbf-9123-148a4bfccf43 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:52.607678 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be706e4a-7cc1-4bbf-9123-148a4bfccf43 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:52.608025 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be706e4a-7cc1-4bbf-9123-148a4bfccf43 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:52.608748 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be706e4a-7cc1-4bbf-9123-148a4bfccf43 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:52.610868 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be706e4a-7cc1-4bbf-9123-148a4bfccf43 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:52.620108 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1237/2450] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 16:22:52 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 16:22:52.620898 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7eacf82-8129-41d3-abde-47abf19a2ab4 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:52.626200 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1214/2451] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:22:52 2022] GET /identity/v3/roles => generated 1427 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:22:52.627189 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be706e4a-7cc1-4bbf-9123-148a4bfccf43 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:52.632586 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7eacf82-8129-41d3-abde-47abf19a2ab4 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:52.639353 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ce4902-029b-47db-8b62-e87dfe37678f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:52.639783 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be706e4a-7cc1-4bbf-9123-148a4bfccf43 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:52.640001 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ce4902-029b-47db-8b62-e87dfe37678f admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:52.640287 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ce4902-029b-47db-8b62-e87dfe37678f admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:52.640565 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ce4902-029b-47db-8b62-e87dfe37678f admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:52.641282 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ce4902-029b-47db-8b62-e87dfe37678f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:52.643139 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ce4902-029b-47db-8b62-e87dfe37678f admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:52.645915 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8a3e13-b401-4b45-8037-2ba90d7b090e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:52.646570 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8a3e13-b401-4b45-8037-2ba90d7b090e admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:52.646843 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8a3e13-b401-4b45-8037-2ba90d7b090e admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:52.647115 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8a3e13-b401-4b45-8037-2ba90d7b090e admin admin] PATH_INFO: `/v3/projects/237c6208e1b9497695df84c32795ec54/users/09b53966f89c4f0d824e24f9a9450ac8/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:52.665291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da8a3e13-b401-4b45-8037-2ba90d7b090e admin admin] RBAC: Authorizing `identity:create_grant(project_id=237c6208e1b9497695df84c32795ec54, user_id=09b53966f89c4f0d824e24f9a9450ac8, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:52.668033 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da8a3e13-b401-4b45-8037-2ba90d7b090e admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:52.681080 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1238/2452] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:22:52 2022] POST /identity/v3/projects => generated 404 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:22:52.682065 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ce4902-029b-47db-8b62-e87dfe37678f admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:52.683446 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1215/2453] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:22:52 2022] PUT /identity/v3/projects/237c6208e1b9497695df84c32795ec54/users/09b53966f89c4f0d824e24f9a9450ac8/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:22:52.693837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90e7e71-5120-43cd-8d7a-a92a2fe3119f None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:52.694128 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90e7e71-5120-43cd-8d7a-a92a2fe3119f None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:52.694407 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90e7e71-5120-43cd-8d7a-a92a2fe3119f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:52.726154 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-d90e7e71-5120-43cd-8d7a-a92a2fe3119f None None] MFA Rules not processed for user `09b53966f89c4f0d824e24f9a9450ac8`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:52.727805 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-d90e7e71-5120-43cd-8d7a-a92a2fe3119f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:52.768371 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ce4902-029b-47db-8b62-e87dfe37678f admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:52.771833 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f67cc9-8ed2-4b35-9370-db01587dc28d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:52.772457 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f67cc9-8ed2-4b35-9370-db01587dc28d admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:52.772742 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f67cc9-8ed2-4b35-9370-db01587dc28d admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:52.773013 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f67cc9-8ed2-4b35-9370-db01587dc28d admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:52.773659 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f67cc9-8ed2-4b35-9370-db01587dc28d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:52.775728 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f67cc9-8ed2-4b35-9370-db01587dc28d admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:52.800645 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1239/2454] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:22:52 2022] POST /identity/v3/users => generated 303 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:22:52.801794 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f67cc9-8ed2-4b35-9370-db01587dc28d admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:52.803425 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-28f67cc9-8ed2-4b35-9370-db01587dc28d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:52.818757 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1216/2455] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:52 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 16:22:52.819810 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90e7e71-5120-43cd-8d7a-a92a2fe3119f None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:52.835686 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90e7e71-5120-43cd-8d7a-a92a2fe3119f None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:52.842785 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d440e2-31dc-474f-b05e-0df4b189d0d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:52.843465 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d440e2-31dc-474f-b05e-0df4b189d0d5 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:52.843833 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d440e2-31dc-474f-b05e-0df4b189d0d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:52.844141 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d440e2-31dc-474f-b05e-0df4b189d0d5 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:52.844924 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d440e2-31dc-474f-b05e-0df4b189d0d5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:52.846721 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d440e2-31dc-474f-b05e-0df4b189d0d5 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:52.858095 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1217/2456] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:22:52 2022] GET /identity/v3/roles => generated 1427 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:22:52.858647 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d440e2-31dc-474f-b05e-0df4b189d0d5 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:52.871273 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d440e2-31dc-474f-b05e-0df4b189d0d5 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:52.876398 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67363d50-d8fc-4e34-a5df-78e2dc9993a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:52.877263 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67363d50-d8fc-4e34-a5df-78e2dc9993a3 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:52.877384 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67363d50-d8fc-4e34-a5df-78e2dc9993a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:52.877669 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67363d50-d8fc-4e34-a5df-78e2dc9993a3 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:52.878298 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67363d50-d8fc-4e34-a5df-78e2dc9993a3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:52.880312 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67363d50-d8fc-4e34-a5df-78e2dc9993a3 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:52.894717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f67cc9-8ed2-4b35-9370-db01587dc28d admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:52.899263 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e43ea2e-08ab-432f-b235-51639ef35c92 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:52.899930 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e43ea2e-08ab-432f-b235-51639ef35c92 service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:52.900378 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e43ea2e-08ab-432f-b235-51639ef35c92 service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:52.900511 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e43ea2e-08ab-432f-b235-51639ef35c92 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:52.903608 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-4e43ea2e-08ab-432f-b235-51639ef35c92 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:52.914155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1218/2457] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:22:52 2022] POST /identity/v3/projects => generated 382 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:22:52.915280 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67363d50-d8fc-4e34-a5df-78e2dc9993a3 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:52.924241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=90,keystone:DELETE=5,keystone:INSERT=5 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:52.973680 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:52.989167 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67363d50-d8fc-4e34-a5df-78e2dc9993a3 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:52.994890 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52710bcf-f7e7-44ad-bc9a-f281b4cac9d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:52.995662 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52710bcf-f7e7-44ad-bc9a-f281b4cac9d9 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:52.996012 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52710bcf-f7e7-44ad-bc9a-f281b4cac9d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:52.996295 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e43ea2e-08ab-432f-b235-51639ef35c92 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:52.996813 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52710bcf-f7e7-44ad-bc9a-f281b4cac9d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:52.997670 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52710bcf-f7e7-44ad-bc9a-f281b4cac9d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:52.998077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e43ea2e-08ab-432f-b235-51639ef35c92 service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:52.999466 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52710bcf-f7e7-44ad-bc9a-f281b4cac9d9 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:53.011470 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1219/2458] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:22:52 2022] GET /identity/v3/roles => generated 1427 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:22:53.013343 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52710bcf-f7e7-44ad-bc9a-f281b4cac9d9 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:53.019485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1240/2459] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:22:52 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 218 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:22:53.020380 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e43ea2e-08ab-432f-b235-51639ef35c92 service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:53.035880 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52710bcf-f7e7-44ad-bc9a-f281b4cac9d9 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:53.036242 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e43ea2e-08ab-432f-b235-51639ef35c92 service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:53.036336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3d70a3-4ac3-4b7f-8b26-891b8da9b172 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:53.036336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3d70a3-4ac3-4b7f-8b26-891b8da9b172 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:53.036336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3d70a3-4ac3-4b7f-8b26-891b8da9b172 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:53.036560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3d70a3-4ac3-4b7f-8b26-891b8da9b172 admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:53.038123 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3d70a3-4ac3-4b7f-8b26-891b8da9b172 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:53.038350 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c4f73e-d02f-4cfe-b12c-c31d59cdca6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:53.039039 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c4f73e-d02f-4cfe-b12c-c31d59cdca6c admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:53.039240 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3d70a3-4ac3-4b7f-8b26-891b8da9b172 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:53.039345 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c4f73e-d02f-4cfe-b12c-c31d59cdca6c admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:53.039581 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c4f73e-d02f-4cfe-b12c-c31d59cdca6c admin admin] PATH_INFO: `/v3/projects/41501d48342648c0a5ff2b239c9c2d9a/users/9570ddfbed254ae2aa942b99ff0f3cff/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:53.056273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c4f73e-d02f-4cfe-b12c-c31d59cdca6c admin admin] RBAC: Authorizing `identity:create_grant(project_id=41501d48342648c0a5ff2b239c9c2d9a, user_id=9570ddfbed254ae2aa942b99ff0f3cff, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:53.059133 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c4f73e-d02f-4cfe-b12c-c31d59cdca6c admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:53.068149 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1220/2460] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:22:53 2022] POST /identity/v3/users => generated 298 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:22:53.072013 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3d70a3-4ac3-4b7f-8b26-891b8da9b172 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:53.078806 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1241/2461] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:22:53 2022] PUT /identity/v3/projects/41501d48342648c0a5ff2b239c9c2d9a/users/9570ddfbed254ae2aa942b99ff0f3cff/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:22:53.089795 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd527f50-6564-4aca-b1e2-10e98bb8b590 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:53.090836 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd527f50-6564-4aca-b1e2-10e98bb8b590 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:53.091486 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd527f50-6564-4aca-b1e2-10e98bb8b590 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:53.094276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3d70a3-4ac3-4b7f-8b26-891b8da9b172 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:53.146785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-cd527f50-6564-4aca-b1e2-10e98bb8b590 None None] MFA Rules not processed for user `9570ddfbed254ae2aa942b99ff0f3cff`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:53.147994 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-cd527f50-6564-4aca-b1e2-10e98bb8b590 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:53.160293 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c32ae1b-f318-4d83-9028-3de29708a5c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:53.160901 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c32ae1b-f318-4d83-9028-3de29708a5c6 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:53.161181 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c32ae1b-f318-4d83-9028-3de29708a5c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:53.161459 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c32ae1b-f318-4d83-9028-3de29708a5c6 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:53.162109 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c32ae1b-f318-4d83-9028-3de29708a5c6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:53.163791 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c32ae1b-f318-4d83-9028-3de29708a5c6 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:53.174052 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1221/2462] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:22:53 2022] GET /identity/v3/roles => generated 1427 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 16:22:53.178234 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c32ae1b-f318-4d83-9028-3de29708a5c6 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:53.188950 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c32ae1b-f318-4d83-9028-3de29708a5c6 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:53.193951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b705a0-3a20-4214-ad12-be989a73b564 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:53.194595 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b705a0-3a20-4214-ad12-be989a73b564 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:53.194901 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b705a0-3a20-4214-ad12-be989a73b564 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:53.195170 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b705a0-3a20-4214-ad12-be989a73b564 admin admin] PATH_INFO: `/v3/projects/1bd763638ed94037af1afa83391edacb/users/4f27c2bd2a2641aca2f15d3bb222443c/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:53.212392 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b705a0-3a20-4214-ad12-be989a73b564 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1bd763638ed94037af1afa83391edacb, user_id=4f27c2bd2a2641aca2f15d3bb222443c, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:53.215928 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b705a0-3a20-4214-ad12-be989a73b564 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:53.218594 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1242/2463] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:53 2022] POST /identity/v3/auth/tokens => generated 3290 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:22:53.223270 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd527f50-6564-4aca-b1e2-10e98bb8b590 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:53.229813 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1222/2464] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:22:53 2022] PUT /identity/v3/projects/1bd763638ed94037af1afa83391edacb/users/4f27c2bd2a2641aca2f15d3bb222443c/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:22:53.233844 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b705a0-3a20-4214-ad12-be989a73b564 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:53.234545 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd527f50-6564-4aca-b1e2-10e98bb8b590 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:53.294887 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562aca48-1482-4feb-9f07-9c632d04542c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:53.295498 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562aca48-1482-4feb-9f07-9c632d04542c admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:53.295820 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562aca48-1482-4feb-9f07-9c632d04542c admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:53.296090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562aca48-1482-4feb-9f07-9c632d04542c admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:53.296723 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-562aca48-1482-4feb-9f07-9c632d04542c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:53.298627 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-562aca48-1482-4feb-9f07-9c632d04542c admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:53.299938 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b705a0-3a20-4214-ad12-be989a73b564 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:53.304887 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6db480-7267-4291-ba3b-9c49c6c7f023 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:53.305489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6db480-7267-4291-ba3b-9c49c6c7f023 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:53.305775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6db480-7267-4291-ba3b-9c49c6c7f023 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:53.306047 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6db480-7267-4291-ba3b-9c49c6c7f023 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:53.306689 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6db480-7267-4291-ba3b-9c49c6c7f023 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:53.308179 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:53.308341 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6db480-7267-4291-ba3b-9c49c6c7f023 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:53.319757 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1223/2465] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:22:53 2022] GET /identity/v3/roles => generated 1427 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 16:22:53.324159 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6db480-7267-4291-ba3b-9c49c6c7f023 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:53.329898 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1243/2466] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:22:53 2022] POST /identity/v3/projects => generated 404 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 16:22:53.333913 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562aca48-1482-4feb-9f07-9c632d04542c admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:53.335184 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6db480-7267-4291-ba3b-9c49c6c7f023 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:53.367050 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:53.394386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a11404e-5668-447b-8d83-d5601b84591b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:53.395109 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a11404e-5668-447b-8d83-d5601b84591b admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:53.395389 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a11404e-5668-447b-8d83-d5601b84591b admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:53.395689 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a11404e-5668-447b-8d83-d5601b84591b admin admin] PATH_INFO: `/v3/projects/1bd763638ed94037af1afa83391edacb/users/4f27c2bd2a2641aca2f15d3bb222443c/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:53.397696 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a11404e-5668-447b-8d83-d5601b84591b admin admin] RBAC: Authorizing `identity:create_grant(project_id=1bd763638ed94037af1afa83391edacb, user_id=4f27c2bd2a2641aca2f15d3bb222443c, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:53.398930 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562aca48-1482-4feb-9f07-9c632d04542c admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:53.400502 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a11404e-5668-447b-8d83-d5601b84591b admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:53.404337 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11175919-b44b-4dcd-ba8f-ef06d567cfd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:53.404956 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11175919-b44b-4dcd-ba8f-ef06d567cfd9 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:53.405379 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11175919-b44b-4dcd-ba8f-ef06d567cfd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:53.405685 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11175919-b44b-4dcd-ba8f-ef06d567cfd9 admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:53.406400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11175919-b44b-4dcd-ba8f-ef06d567cfd9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:53.408392 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11175919-b44b-4dcd-ba8f-ef06d567cfd9 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:53.411231 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1224/2467] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:22:53 2022] PUT /identity/v3/projects/1bd763638ed94037af1afa83391edacb/users/4f27c2bd2a2641aca2f15d3bb222443c/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 88 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:22:53.422668 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3926ac63-47ab-45e5-8bd5-81d396abcbfe None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:53.422991 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3926ac63-47ab-45e5-8bd5-81d396abcbfe None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:53.423261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3926ac63-47ab-45e5-8bd5-81d396abcbfe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:53.430117 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1244/2468] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:22:53 2022] POST /identity/v3/users => generated 309 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 16:22:53.434784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11175919-b44b-4dcd-ba8f-ef06d567cfd9 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:53.444851 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11175919-b44b-4dcd-ba8f-ef06d567cfd9 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:53.454276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-3926ac63-47ab-45e5-8bd5-81d396abcbfe None None] MFA Rules not processed for user `4f27c2bd2a2641aca2f15d3bb222443c`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:53.455508 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-3926ac63-47ab-45e5-8bd5-81d396abcbfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:53.505179 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9ba67e-7801-4e2b-a5bb-826e3f9b8052 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:53.505793 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9ba67e-7801-4e2b-a5bb-826e3f9b8052 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:53.506076 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9ba67e-7801-4e2b-a5bb-826e3f9b8052 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:53.506352 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9ba67e-7801-4e2b-a5bb-826e3f9b8052 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:53.506996 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9ba67e-7801-4e2b-a5bb-826e3f9b8052 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:53.508638 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9ba67e-7801-4e2b-a5bb-826e3f9b8052 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:53.518439 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1245/2469] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:22:53 2022] GET /identity/v3/roles => generated 1427 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 16:22:53.522590 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9ba67e-7801-4e2b-a5bb-826e3f9b8052 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:53.526595 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1225/2470] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:53 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 16:22:53.533887 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9ba67e-7801-4e2b-a5bb-826e3f9b8052 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:53.537892 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3926ac63-47ab-45e5-8bd5-81d396abcbfe None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:53.538684 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a6986e-d4bc-40c6-9162-62a0476713cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:53.539346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a6986e-d4bc-40c6-9162-62a0476713cc admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:53.539665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a6986e-d4bc-40c6-9162-62a0476713cc admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:53.539939 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a6986e-d4bc-40c6-9162-62a0476713cc admin admin] PATH_INFO: `/v3/projects/9da9f444153e4dcc8d6be0bd9ccb64f0/users/ffa1ff1936d942db88160c3f4df5cc2e/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:53.555788 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1a6986e-d4bc-40c6-9162-62a0476713cc admin admin] RBAC: Authorizing `identity:create_grant(project_id=9da9f444153e4dcc8d6be0bd9ccb64f0, user_id=ffa1ff1936d942db88160c3f4df5cc2e, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:53.558466 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1a6986e-d4bc-40c6-9162-62a0476713cc admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:53.573204 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1246/2471] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:22:53 2022] PUT /identity/v3/projects/9da9f444153e4dcc8d6be0bd9ccb64f0/users/ffa1ff1936d942db88160c3f4df5cc2e/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:22:53.577735 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a6986e-d4bc-40c6-9162-62a0476713cc admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:53.596300 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3926ac63-47ab-45e5-8bd5-81d396abcbfe None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:53.634027 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:53.646845 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a6986e-d4bc-40c6-9162-62a0476713cc admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:53.648706 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e569e511-a0db-490e-a526-e157e28ec2a2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:53.649258 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e569e511-a0db-490e-a526-e157e28ec2a2 service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:53.649538 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e569e511-a0db-490e-a526-e157e28ec2a2 service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:53.649814 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e569e511-a0db-490e-a526-e157e28ec2a2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:53.651853 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d96243-e255-4465-a544-38a5d180e0ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:53.652630 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d96243-e255-4465-a544-38a5d180e0ba admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:53.652940 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d96243-e255-4465-a544-38a5d180e0ba admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:53.653224 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d96243-e255-4465-a544-38a5d180e0ba admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:53.653904 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d96243-e255-4465-a544-38a5d180e0ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:53.655508 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d96243-e255-4465-a544-38a5d180e0ba admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:53.665569 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1247/2472] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:22:53 2022] GET /identity/v3/roles => generated 1427 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 16:22:53.670789 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d96243-e255-4465-a544-38a5d180e0ba admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:53.678045 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:53.680583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d96243-e255-4465-a544-38a5d180e0ba admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:53.686729 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2202c9b6-775a-4ecd-8062-585bdefd9711 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:53.687386 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2202c9b6-775a-4ecd-8062-585bdefd9711 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:53.687704 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2202c9b6-775a-4ecd-8062-585bdefd9711 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:53.687993 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2202c9b6-775a-4ecd-8062-585bdefd9711 admin admin] PATH_INFO: `/v3/projects/9da9f444153e4dcc8d6be0bd9ccb64f0/users/ffa1ff1936d942db88160c3f4df5cc2e/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:53.689939 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2202c9b6-775a-4ecd-8062-585bdefd9711 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9da9f444153e4dcc8d6be0bd9ccb64f0, user_id=ffa1ff1936d942db88160c3f4df5cc2e, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:53.692634 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2202c9b6-775a-4ecd-8062-585bdefd9711 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:53.704184 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1248/2473] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:22:53 2022] PUT /identity/v3/projects/9da9f444153e4dcc8d6be0bd9ccb64f0/users/ffa1ff1936d942db88160c3f4df5cc2e/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:22:53.716122 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db383c0-4b8f-4da7-81ad-190ad4ebd679 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:53.716419 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db383c0-4b8f-4da7-81ad-190ad4ebd679 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:53.716695 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db383c0-4b8f-4da7-81ad-190ad4ebd679 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:53.719066 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e569e511-a0db-490e-a526-e157e28ec2a2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:53.720623 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e569e511-a0db-490e-a526-e157e28ec2a2 service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:53.730411 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1226/2474] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:22:53 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 193 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 16:22:53.747241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-6db383c0-4b8f-4da7-81ad-190ad4ebd679 None None] MFA Rules not processed for user `ffa1ff1936d942db88160c3f4df5cc2e`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:53.748488 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-6db383c0-4b8f-4da7-81ad-190ad4ebd679 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:53.826306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1249/2475] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:53 2022] POST /identity/v3/auth/tokens => generated 3357 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 16:22:53.837412 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e569e511-a0db-490e-a526-e157e28ec2a2 service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:53.847439 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e569e511-a0db-490e-a526-e157e28ec2a2 service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:53.901954 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d226c20-0034-49f8-ba39-b863f1e3f71c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:53.903004 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d226c20-0034-49f8-ba39-b863f1e3f71c service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:53.903350 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d226c20-0034-49f8-ba39-b863f1e3f71c service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:53.903624 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d226c20-0034-49f8-ba39-b863f1e3f71c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:53.913565 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d226c20-0034-49f8-ba39-b863f1e3f71c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:53.915121 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d226c20-0034-49f8-ba39-b863f1e3f71c service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:53.925533 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1227/2476] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:22:53 2022] GET /identity/v3/auth/tokens => generated 3357 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:22:54.171440 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db383c0-4b8f-4da7-81ad-190ad4ebd679 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:54.253536 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db383c0-4b8f-4da7-81ad-190ad4ebd679 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:54.258033 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622a1ec6-0b28-442c-9575-3beef3581bb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:54.258655 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622a1ec6-0b28-442c-9575-3beef3581bb5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:54.258939 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622a1ec6-0b28-442c-9575-3beef3581bb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:54.259216 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622a1ec6-0b28-442c-9575-3beef3581bb5 admin admin] PATH_INFO: `/v3/users/09b53966f89c4f0d824e24f9a9450ac8` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:54.260376 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622a1ec6-0b28-442c-9575-3beef3581bb5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=09b53966f89c4f0d824e24f9a9450ac8)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:54.262338 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622a1ec6-0b28-442c-9575-3beef3581bb5 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:54.311391 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-622a1ec6-0b28-442c-9575-3beef3581bb5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '09b53966f89c4f0d824e24f9a9450ac8'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:54.312126 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-622a1ec6-0b28-442c-9575-3beef3581bb5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '09b53966f89c4f0d824e24f9a9450ac8'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:54.312782 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-622a1ec6-0b28-442c-9575-3beef3581bb5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '09b53966f89c4f0d824e24f9a9450ac8'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:54.320084 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-622a1ec6-0b28-442c-9575-3beef3581bb5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '09b53966f89c4f0d824e24f9a9450ac8'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:54.325987 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=89,keystone:INSERT=2,keystone:DELETE=9 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:54.334343 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-622a1ec6-0b28-442c-9575-3beef3581bb5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '09b53966f89c4f0d824e24f9a9450ac8'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:54.347240 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1250/2477] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:22:54 2022] DELETE /identity/v3/users/09b53966f89c4f0d824e24f9a9450ac8 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:22:54.351824 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b71458-9893-462b-85d3-5c3df8a0712f None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:54.352287 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b71458-9893-462b-85d3-5c3df8a0712f None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:54.352404 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b71458-9893-462b-85d3-5c3df8a0712f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:54.371265 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-10b71458-9893-462b-85d3-5c3df8a0712f None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:54.373172 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-10b71458-9893-462b-85d3-5c3df8a0712f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:54.432044 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1228/2478] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:54 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:22:54.442628 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622a1ec6-0b28-442c-9575-3beef3581bb5 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:54.444063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-622a1ec6-0b28-442c-9575-3beef3581bb5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:54.503258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622a1ec6-0b28-442c-9575-3beef3581bb5 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:54.506944 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22b4b1d-5f54-401a-8b71-1708a64ae0c5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:54.507545 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22b4b1d-5f54-401a-8b71-1708a64ae0c5 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:54.507860 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22b4b1d-5f54-401a-8b71-1708a64ae0c5 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:54.508136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22b4b1d-5f54-401a-8b71-1708a64ae0c5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:54.517479 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d22b4b1d-5f54-401a-8b71-1708a64ae0c5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:54.518913 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d22b4b1d-5f54-401a-8b71-1708a64ae0c5 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:54.528622 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1251/2479] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:22:54 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:22:54.792979 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b71458-9893-462b-85d3-5c3df8a0712f None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:54.795599 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-10b71458-9893-462b-85d3-5c3df8a0712f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:54.808357 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b71458-9893-462b-85d3-5c3df8a0712f None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:54.813406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ecf10d-f803-4554-a7f0-4ab4f94491fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:54.814069 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ecf10d-f803-4554-a7f0-4ab4f94491fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:54.814348 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ecf10d-f803-4554-a7f0-4ab4f94491fc admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:54.814622 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ecf10d-f803-4554-a7f0-4ab4f94491fc admin admin] PATH_INFO: `/v3/projects/237c6208e1b9497695df84c32795ec54` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:54.815908 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ecf10d-f803-4554-a7f0-4ab4f94491fc admin admin] RBAC: Authorizing `identity:delete_project(project_id=237c6208e1b9497695df84c32795ec54)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:54.817917 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ecf10d-f803-4554-a7f0-4ab4f94491fc admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:54.863679 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22b4b1d-5f54-401a-8b71-1708a64ae0c5 service neutron] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:54.865062 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-d22b4b1d-5f54-401a-8b71-1708a64ae0c5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:54.878607 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22b4b1d-5f54-401a-8b71-1708a64ae0c5 service neutron] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:54.878813 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-15ecf10d-f803-4554-a7f0-4ab4f94491fc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '237c6208e1b9497695df84c32795ec54', 'request_id': 'req-15ecf10d-f803-4554-a7f0-4ab4f94491fc'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:54.885285 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-15ecf10d-f803-4554-a7f0-4ab4f94491fc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '237c6208e1b9497695df84c32795ec54', 'request_id': 'req-15ecf10d-f803-4554-a7f0-4ab4f94491fc'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:54.890119 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:54.893060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-15ecf10d-f803-4554-a7f0-4ab4f94491fc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '237c6208e1b9497695df84c32795ec54', 'request_id': 'req-15ecf10d-f803-4554-a7f0-4ab4f94491fc'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:54.908554 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-15ecf10d-f803-4554-a7f0-4ab4f94491fc admin admin] The token cache is being invalidate because project 237c6208e1b9497695df84c32795ec54 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:22:54.908847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-15ecf10d-f803-4554-a7f0-4ab4f94491fc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:54.909525 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-15ecf10d-f803-4554-a7f0-4ab4f94491fc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:54.911017 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1229/2480] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:22:54 2022] DELETE /identity/v3/projects/237c6208e1b9497695df84c32795ec54 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:22:54.915345 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ecf10d-f803-4554-a7f0-4ab4f94491fc admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:54.916740 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-15ecf10d-f803-4554-a7f0-4ab4f94491fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:54.938224 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511ac30b-b6cc-4a0f-80c2-108cdf0bf740 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:54.938861 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511ac30b-b6cc-4a0f-80c2-108cdf0bf740 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:54.939144 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511ac30b-b6cc-4a0f-80c2-108cdf0bf740 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:54.939429 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511ac30b-b6cc-4a0f-80c2-108cdf0bf740 admin admin] PATH_INFO: `/v3/users/9570ddfbed254ae2aa942b99ff0f3cff` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:54.940692 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511ac30b-b6cc-4a0f-80c2-108cdf0bf740 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9570ddfbed254ae2aa942b99ff0f3cff)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:54.942634 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511ac30b-b6cc-4a0f-80c2-108cdf0bf740 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:54.988052 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ecf10d-f803-4554-a7f0-4ab4f94491fc admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:54.991992 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-511ac30b-b6cc-4a0f-80c2-108cdf0bf740 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9570ddfbed254ae2aa942b99ff0f3cff'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:54.992653 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-511ac30b-b6cc-4a0f-80c2-108cdf0bf740 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9570ddfbed254ae2aa942b99ff0f3cff'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:54.993274 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-511ac30b-b6cc-4a0f-80c2-108cdf0bf740 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9570ddfbed254ae2aa942b99ff0f3cff'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:54.998372 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0d134a-75dd-469b-8cd4-335982a09dce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:54.999001 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0d134a-75dd-469b-8cd4-335982a09dce admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:54.999281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0d134a-75dd-469b-8cd4-335982a09dce admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:54.999688 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0d134a-75dd-469b-8cd4-335982a09dce admin admin] PATH_INFO: `/v3/users/4f27c2bd2a2641aca2f15d3bb222443c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:55.000764 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0d134a-75dd-469b-8cd4-335982a09dce admin admin] RBAC: Authorizing `identity:delete_user(user_id=4f27c2bd2a2641aca2f15d3bb222443c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:55.001972 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-511ac30b-b6cc-4a0f-80c2-108cdf0bf740 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9570ddfbed254ae2aa942b99ff0f3cff'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:55.002690 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0d134a-75dd-469b-8cd4-335982a09dce admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:55.015315 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-511ac30b-b6cc-4a0f-80c2-108cdf0bf740 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9570ddfbed254ae2aa942b99ff0f3cff'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:55.023813 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:55.031074 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1252/2481] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:22:54 2022] DELETE /identity/v3/users/9570ddfbed254ae2aa942b99ff0f3cff => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:22:55.036129 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1ae4b5-c2f3-4591-b3cf-0a8110140ab9 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:55.036484 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1ae4b5-c2f3-4591-b3cf-0a8110140ab9 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:55.037293 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1ae4b5-c2f3-4591-b3cf-0a8110140ab9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:55.054940 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-db0d134a-75dd-469b-8cd4-335982a09dce admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4f27c2bd2a2641aca2f15d3bb222443c'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:55.055657 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-db0d134a-75dd-469b-8cd4-335982a09dce admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4f27c2bd2a2641aca2f15d3bb222443c'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:55.059323 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-2a1ae4b5-c2f3-4591-b3cf-0a8110140ab9 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:55.060709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-2a1ae4b5-c2f3-4591-b3cf-0a8110140ab9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:55.063282 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-db0d134a-75dd-469b-8cd4-335982a09dce admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4f27c2bd2a2641aca2f15d3bb222443c'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:55.075968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-db0d134a-75dd-469b-8cd4-335982a09dce admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4f27c2bd2a2641aca2f15d3bb222443c'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:55.076628 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-db0d134a-75dd-469b-8cd4-335982a09dce admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4f27c2bd2a2641aca2f15d3bb222443c'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:55.091860 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1230/2482] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:22:54 2022] DELETE /identity/v3/users/4f27c2bd2a2641aca2f15d3bb222443c => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:22:55.122400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1253/2483] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:55 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:22:55.132179 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0d134a-75dd-469b-8cd4-335982a09dce admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:55.133622 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-db0d134a-75dd-469b-8cd4-335982a09dce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:55.197356 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0d134a-75dd-469b-8cd4-335982a09dce admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:55.202403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5240d155-e90d-498b-b2d8-60ae91d2b385 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:55.203203 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5240d155-e90d-498b-b2d8-60ae91d2b385 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:55.203489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5240d155-e90d-498b-b2d8-60ae91d2b385 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:55.203865 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5240d155-e90d-498b-b2d8-60ae91d2b385 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:55.231061 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:DELETE=12,keystone:SELECT=86,keystone:INSERT=2 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:55.295266 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5240d155-e90d-498b-b2d8-60ae91d2b385 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:55.296790 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5240d155-e90d-498b-b2d8-60ae91d2b385 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:55.311803 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1231/2484] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:22:55 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:22:55.482751 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1ae4b5-c2f3-4591-b3cf-0a8110140ab9 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:55.484486 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-2a1ae4b5-c2f3-4591-b3cf-0a8110140ab9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:55.497954 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1ae4b5-c2f3-4591-b3cf-0a8110140ab9 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:55.503756 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c1f3f7-e437-45de-9107-166aee7536cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:55.504412 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c1f3f7-e437-45de-9107-166aee7536cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:55.504755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c1f3f7-e437-45de-9107-166aee7536cf admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:55.505041 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c1f3f7-e437-45de-9107-166aee7536cf admin admin] PATH_INFO: `/v3/projects/1bd763638ed94037af1afa83391edacb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:55.506255 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c1f3f7-e437-45de-9107-166aee7536cf admin admin] RBAC: Authorizing `identity:delete_project(project_id=1bd763638ed94037af1afa83391edacb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:55.508314 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c1f3f7-e437-45de-9107-166aee7536cf admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:55.571393 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-13c1f3f7-e437-45de-9107-166aee7536cf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1bd763638ed94037af1afa83391edacb', 'request_id': 'req-13c1f3f7-e437-45de-9107-166aee7536cf'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:55.579297 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-13c1f3f7-e437-45de-9107-166aee7536cf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1bd763638ed94037af1afa83391edacb', 'request_id': 'req-13c1f3f7-e437-45de-9107-166aee7536cf'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:55.585338 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-13c1f3f7-e437-45de-9107-166aee7536cf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1bd763638ed94037af1afa83391edacb', 'request_id': 'req-13c1f3f7-e437-45de-9107-166aee7536cf'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:55.599067 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-13c1f3f7-e437-45de-9107-166aee7536cf admin admin] The token cache is being invalidate because project 1bd763638ed94037af1afa83391edacb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:22:55.599358 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-13c1f3f7-e437-45de-9107-166aee7536cf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:55.600000 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-13c1f3f7-e437-45de-9107-166aee7536cf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:55.601503 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1254/2485] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:22:55 2022] DELETE /identity/v3/projects/1bd763638ed94037af1afa83391edacb => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:22:55.607111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47aa349f-e1e8-4009-abdb-22202a580b59 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:55.607582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47aa349f-e1e8-4009-abdb-22202a580b59 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:55.607720 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47aa349f-e1e8-4009-abdb-22202a580b59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:55.629929 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-47aa349f-e1e8-4009-abdb-22202a580b59 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:55.631235 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-47aa349f-e1e8-4009-abdb-22202a580b59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:55.677706 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c1f3f7-e437-45de-9107-166aee7536cf admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:55.679834 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-13c1f3f7-e437-45de-9107-166aee7536cf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:55.704107 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1232/2486] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:55 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 16:22:55.712099 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a17ba69-87ad-4143-ae8d-b5ec3e5a9db3 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:55.712421 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a17ba69-87ad-4143-ae8d-b5ec3e5a9db3 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:55.712949 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a17ba69-87ad-4143-ae8d-b5ec3e5a9db3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:55.734571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-4a17ba69-87ad-4143-ae8d-b5ec3e5a9db3 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:55.736062 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-4a17ba69-87ad-4143-ae8d-b5ec3e5a9db3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:55.744172 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1233/2487] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:55 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:22:55.749218 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a17ba69-87ad-4143-ae8d-b5ec3e5a9db3 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:55.750677 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c1f3f7-e437-45de-9107-166aee7536cf admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:55.755423 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ed56c4-5e82-4d3b-b600-d35602db1101 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:55.756097 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ed56c4-5e82-4d3b-b600-d35602db1101 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:55.756384 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ed56c4-5e82-4d3b-b600-d35602db1101 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:55.756663 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ed56c4-5e82-4d3b-b600-d35602db1101 admin admin] PATH_INFO: `/v3/projects/41501d48342648c0a5ff2b239c9c2d9a` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:55.757814 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ed56c4-5e82-4d3b-b600-d35602db1101 admin admin] RBAC: Authorizing `identity:delete_project(project_id=41501d48342648c0a5ff2b239c9c2d9a)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:55.758404 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a17ba69-87ad-4143-ae8d-b5ec3e5a9db3 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:55.759787 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ed56c4-5e82-4d3b-b600-d35602db1101 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:55.763742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9c569a-1ccb-4f70-b6e3-2daf77144b21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:55.764342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9c569a-1ccb-4f70-b6e3-2daf77144b21 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:55.764621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9c569a-1ccb-4f70-b6e3-2daf77144b21 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:55.764898 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9c569a-1ccb-4f70-b6e3-2daf77144b21 admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:55.765558 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9c569a-1ccb-4f70-b6e3-2daf77144b21 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 16:22:55.765859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9c569a-1ccb-4f70-b6e3-2daf77144b21 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:55.767440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9c569a-1ccb-4f70-b6e3-2daf77144b21 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:55.782158 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1234/2488] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 16:22:55 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 16:22:55.786654 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9c569a-1ccb-4f70-b6e3-2daf77144b21 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:55.795954 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:55.797466 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9c569a-1ccb-4f70-b6e3-2daf77144b21 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:55.801977 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028ca79e-5cb8-4ca0-bcf4-c8ac6d5c9983 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:55.802666 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028ca79e-5cb8-4ca0-bcf4-c8ac6d5c9983 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:55.802949 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028ca79e-5cb8-4ca0-bcf4-c8ac6d5c9983 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:55.803148 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028ca79e-5cb8-4ca0-bcf4-c8ac6d5c9983 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:55.803869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-028ca79e-5cb8-4ca0-bcf4-c8ac6d5c9983 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:55.805718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-028ca79e-5cb8-4ca0-bcf4-c8ac6d5c9983 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:55.815287 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-c9ed56c4-5e82-4d3b-b600-d35602db1101 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '41501d48342648c0a5ff2b239c9c2d9a', 'request_id': 'req-c9ed56c4-5e82-4d3b-b600-d35602db1101'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:55.821552 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-c9ed56c4-5e82-4d3b-b600-d35602db1101 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '41501d48342648c0a5ff2b239c9c2d9a', 'request_id': 'req-c9ed56c4-5e82-4d3b-b600-d35602db1101'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:55.828502 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-c9ed56c4-5e82-4d3b-b600-d35602db1101 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '41501d48342648c0a5ff2b239c9c2d9a', 'request_id': 'req-c9ed56c4-5e82-4d3b-b600-d35602db1101'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:55.844406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1235/2489] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:22:55 2022] POST /identity/v3/projects => generated 374 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 16:22:55.848915 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-c9ed56c4-5e82-4d3b-b600-d35602db1101 admin admin] The token cache is being invalidate because project 41501d48342648c0a5ff2b239c9c2d9a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:22:55.849150 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028ca79e-5cb8-4ca0-bcf4-c8ac6d5c9983 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:55.849317 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-c9ed56c4-5e82-4d3b-b600-d35602db1101 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:55.849916 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-c9ed56c4-5e82-4d3b-b600-d35602db1101 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:55.851335 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-028ca79e-5cb8-4ca0-bcf4-c8ac6d5c9983 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:55.851730 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1255/2490] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:22:55 2022] DELETE /identity/v3/projects/41501d48342648c0a5ff2b239c9c2d9a => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:22:55.856413 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ed56c4-5e82-4d3b-b600-d35602db1101 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:55.868134 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-c9ed56c4-5e82-4d3b-b600-d35602db1101 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:55.932631 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028ca79e-5cb8-4ca0-bcf4-c8ac6d5c9983 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:55.938802 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8dfad6e-94de-42bf-8503-97604a7e0d18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:55.939581 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dfad6e-94de-42bf-8503-97604a7e0d18 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:55.939724 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dfad6e-94de-42bf-8503-97604a7e0d18 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:55.939983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dfad6e-94de-42bf-8503-97604a7e0d18 admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:55.940614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8dfad6e-94de-42bf-8503-97604a7e0d18 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:55.942544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8dfad6e-94de-42bf-8503-97604a7e0d18 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:55.943122 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ed56c4-5e82-4d3b-b600-d35602db1101 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:55.948075 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd46b5c7-68d3-4d3c-920b-f550545e237e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:55.948716 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd46b5c7-68d3-4d3c-920b-f550545e237e admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:55.948999 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd46b5c7-68d3-4d3c-920b-f550545e237e admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:55.949889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd46b5c7-68d3-4d3c-920b-f550545e237e admin admin] PATH_INFO: `/v3/users/ffa1ff1936d942db88160c3f4df5cc2e` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:55.952583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd46b5c7-68d3-4d3c-920b-f550545e237e admin admin] RBAC: Authorizing `identity:delete_user(user_id=ffa1ff1936d942db88160c3f4df5cc2e)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:55.955343 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd46b5c7-68d3-4d3c-920b-f550545e237e admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:55.966161 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1236/2491] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:22:55 2022] POST /identity/v3/users => generated 288 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:22:55.970431 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8dfad6e-94de-42bf-8503-97604a7e0d18 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:55.977196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:55.981756 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8dfad6e-94de-42bf-8503-97604a7e0d18 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:55.988693 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e245ddc-35ef-42de-ae49-e2e57dbe045b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:55.989266 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e245ddc-35ef-42de-ae49-e2e57dbe045b admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:55.989550 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e245ddc-35ef-42de-ae49-e2e57dbe045b admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:55.989822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e245ddc-35ef-42de-ae49-e2e57dbe045b admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:55.990470 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e245ddc-35ef-42de-ae49-e2e57dbe045b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:55.992212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e245ddc-35ef-42de-ae49-e2e57dbe045b admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:56.003024 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1237/2492] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:22:55 2022] GET /identity/v3/roles => generated 1427 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 16:22:56.006958 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e245ddc-35ef-42de-ae49-e2e57dbe045b admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:56.007864 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-dd46b5c7-68d3-4d3c-920b-f550545e237e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ffa1ff1936d942db88160c3f4df5cc2e'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:56.008672 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-dd46b5c7-68d3-4d3c-920b-f550545e237e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ffa1ff1936d942db88160c3f4df5cc2e'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:56.009295 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-dd46b5c7-68d3-4d3c-920b-f550545e237e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ffa1ff1936d942db88160c3f4df5cc2e'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:56.017682 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-dd46b5c7-68d3-4d3c-920b-f550545e237e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ffa1ff1936d942db88160c3f4df5cc2e'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:56.017958 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e245ddc-35ef-42de-ae49-e2e57dbe045b admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:56.022855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f004dc6b-c3bf-4c9a-9acf-aa0d0e1ac0c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:56.023447 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f004dc6b-c3bf-4c9a-9acf-aa0d0e1ac0c9 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:56.023786 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f004dc6b-c3bf-4c9a-9acf-aa0d0e1ac0c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:56.024138 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f004dc6b-c3bf-4c9a-9acf-aa0d0e1ac0c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:56.024786 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f004dc6b-c3bf-4c9a-9acf-aa0d0e1ac0c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:56.026401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f004dc6b-c3bf-4c9a-9acf-aa0d0e1ac0c9 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:56.031657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-dd46b5c7-68d3-4d3c-920b-f550545e237e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ffa1ff1936d942db88160c3f4df5cc2e'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:56.037713 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1238/2493] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:22:56 2022] GET /identity/v3/roles => generated 1427 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 16:22:56.041541 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f004dc6b-c3bf-4c9a-9acf-aa0d0e1ac0c9 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:56.043291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-f004dc6b-c3bf-4c9a-9acf-aa0d0e1ac0c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:56.044485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1256/2494] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:22:55 2022] DELETE /identity/v3/users/ffa1ff1936d942db88160c3f4df5cc2e => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:22:56.056465 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f004dc6b-c3bf-4c9a-9acf-aa0d0e1ac0c9 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:56.120679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9ff974-9376-43b6-9d41-4490f98c821e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:56.121969 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9ff974-9376-43b6-9d41-4490f98c821e admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:56.122393 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9ff974-9376-43b6-9d41-4490f98c821e admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:56.122671 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9ff974-9376-43b6-9d41-4490f98c821e admin admin] PATH_INFO: `/v3/projects/786884b7dee04cb4b508adc4a7e1aca0/users/00c5edf3e19846d4b79318572dd514ae/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:56.139546 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9ff974-9376-43b6-9d41-4490f98c821e admin admin] RBAC: Authorizing `identity:create_grant(project_id=786884b7dee04cb4b508adc4a7e1aca0, user_id=00c5edf3e19846d4b79318572dd514ae, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:56.142233 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9ff974-9376-43b6-9d41-4490f98c821e admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:56.155990 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1239/2495] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:22:56 2022] PUT /identity/v3/projects/786884b7dee04cb4b508adc4a7e1aca0/users/00c5edf3e19846d4b79318572dd514ae/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:22:56.173603 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b500f6-248c-45fa-9690-3f8372d2c59c None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:56.174145 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b500f6-248c-45fa-9690-3f8372d2c59c None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:56.174639 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b500f6-248c-45fa-9690-3f8372d2c59c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:56.206542 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-c4b500f6-248c-45fa-9690-3f8372d2c59c None None] MFA Rules not processed for user `00c5edf3e19846d4b79318572dd514ae`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:56.208378 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-c4b500f6-248c-45fa-9690-3f8372d2c59c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:56.236058 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:DELETE=16,keystone:SELECT=82,keystone:INSERT=2 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:56.293842 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1257/2496] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:56 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 16:22:56.305437 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9ff974-9376-43b6-9d41-4490f98c821e admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:56.307541 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-8a9ff974-9376-43b6-9d41-4490f98c821e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:56.357988 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b500f6-248c-45fa-9690-3f8372d2c59c None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:56.359533 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-c4b500f6-248c-45fa-9690-3f8372d2c59c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:56.387621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9ff974-9376-43b6-9d41-4490f98c821e admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:56.392451 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2291f77c-def1-442c-b4a0-743a6fa8c50f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:56.393403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2291f77c-def1-442c-b4a0-743a6fa8c50f service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:56.393784 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2291f77c-def1-442c-b4a0-743a6fa8c50f service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:56.394152 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2291f77c-def1-442c-b4a0-743a6fa8c50f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:56.404342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2291f77c-def1-442c-b4a0-743a6fa8c50f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:56.405838 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2291f77c-def1-442c-b4a0-743a6fa8c50f service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:56.416153 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1240/2497] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:22:56 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:22:56.443948 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2291f77c-def1-442c-b4a0-743a6fa8c50f service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:56.457351 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b500f6-248c-45fa-9690-3f8372d2c59c None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:56.462359 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2291f77c-def1-442c-b4a0-743a6fa8c50f service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:56.466716 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d66eb8c-b4bf-49e2-87e7-4271fe362c46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:56.467374 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d66eb8c-b4bf-49e2-87e7-4271fe362c46 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:56.467687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d66eb8c-b4bf-49e2-87e7-4271fe362c46 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:56.467972 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d66eb8c-b4bf-49e2-87e7-4271fe362c46 admin admin] PATH_INFO: `/v3/projects/9da9f444153e4dcc8d6be0bd9ccb64f0` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:56.468319 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1682f2-9e87-4f58-8d23-f7bbcb649830 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:56.469003 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1682f2-9e87-4f58-8d23-f7bbcb649830 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:56.469165 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d66eb8c-b4bf-49e2-87e7-4271fe362c46 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9da9f444153e4dcc8d6be0bd9ccb64f0)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:56.469404 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1682f2-9e87-4f58-8d23-f7bbcb649830 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:56.469745 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1682f2-9e87-4f58-8d23-f7bbcb649830 admin admin] PATH_INFO: `/v3/users/00c5edf3e19846d4b79318572dd514ae` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:56.470967 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b1682f2-9e87-4f58-8d23-f7bbcb649830 admin admin] RBAC: Authorizing `identity:delete_user(user_id=00c5edf3e19846d4b79318572dd514ae)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:56.471218 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d66eb8c-b4bf-49e2-87e7-4271fe362c46 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:56.473170 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b1682f2-9e87-4f58-8d23-f7bbcb649830 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:56.527160 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0d66eb8c-b4bf-49e2-87e7-4271fe362c46 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9da9f444153e4dcc8d6be0bd9ccb64f0', 'request_id': 'req-0d66eb8c-b4bf-49e2-87e7-4271fe362c46'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:56.528518 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-8b1682f2-9e87-4f58-8d23-f7bbcb649830 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '00c5edf3e19846d4b79318572dd514ae'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:56.529208 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-8b1682f2-9e87-4f58-8d23-f7bbcb649830 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '00c5edf3e19846d4b79318572dd514ae'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:56.533748 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0d66eb8c-b4bf-49e2-87e7-4271fe362c46 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9da9f444153e4dcc8d6be0bd9ccb64f0', 'request_id': 'req-0d66eb8c-b4bf-49e2-87e7-4271fe362c46'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:56.538524 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-8b1682f2-9e87-4f58-8d23-f7bbcb649830 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '00c5edf3e19846d4b79318572dd514ae'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:56.540318 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0d66eb8c-b4bf-49e2-87e7-4271fe362c46 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9da9f444153e4dcc8d6be0bd9ccb64f0', 'request_id': 'req-0d66eb8c-b4bf-49e2-87e7-4271fe362c46'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:56.548786 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=91,keystone:INSERT=1,keystone:DELETE=8 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:56.555798 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0d66eb8c-b4bf-49e2-87e7-4271fe362c46 admin admin] The token cache is being invalidate because project 9da9f444153e4dcc8d6be0bd9ccb64f0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:22:56.556093 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0d66eb8c-b4bf-49e2-87e7-4271fe362c46 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:56.556931 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0d66eb8c-b4bf-49e2-87e7-4271fe362c46 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:56.558625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1258/2498] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:22:56 2022] DELETE /identity/v3/projects/9da9f444153e4dcc8d6be0bd9ccb64f0 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:22:56.563348 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbcd243-56d9-4387-962c-0dae58a53368 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:56.563662 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbcd243-56d9-4387-962c-0dae58a53368 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:56.563960 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbcd243-56d9-4387-962c-0dae58a53368 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:56.564463 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-8b1682f2-9e87-4f58-8d23-f7bbcb649830 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '00c5edf3e19846d4b79318572dd514ae'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:56.565155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-8b1682f2-9e87-4f58-8d23-f7bbcb649830 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '00c5edf3e19846d4b79318572dd514ae'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:56.585345 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1241/2499] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:22:56 2022] DELETE /identity/v3/users/00c5edf3e19846d4b79318572dd514ae => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:22:56.591272 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cb203c-5066-4aca-9300-b8abe20d896d None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:56.591568 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cb203c-5066-4aca-9300-b8abe20d896d None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:56.591900 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cb203c-5066-4aca-9300-b8abe20d896d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:56.593998 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-7bbcd243-56d9-4387-962c-0dae58a53368 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:56.595356 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-7bbcd243-56d9-4387-962c-0dae58a53368 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:56.612169 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-f6cb203c-5066-4aca-9300-b8abe20d896d None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:56.613359 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-f6cb203c-5066-4aca-9300-b8abe20d896d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:56.650467 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:56.670722 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1259/2500] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:56 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:22:56.678084 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc042aca-0c2e-47ae-ab4d-019bf2aaf617 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:56.678518 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc042aca-0c2e-47ae-ab4d-019bf2aaf617 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:56.678800 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc042aca-0c2e-47ae-ab4d-019bf2aaf617 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:56.688447 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1242/2501] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:56 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:22:56.695405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6cb203c-5066-4aca-9300-b8abe20d896d None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:56.696828 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-f6cb203c-5066-4aca-9300-b8abe20d896d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:56.701708 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-cc042aca-0c2e-47ae-ab4d-019bf2aaf617 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:56.702877 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-cc042aca-0c2e-47ae-ab4d-019bf2aaf617 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:56.711441 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1260/2502] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:56 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:22:56.719111 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc042aca-0c2e-47ae-ab4d-019bf2aaf617 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:56.729728 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc042aca-0c2e-47ae-ab4d-019bf2aaf617 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:56.736605 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7571c5e0-068c-4d67-b439-6ecfc8532a11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:56.737697 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7571c5e0-068c-4d67-b439-6ecfc8532a11 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:56.738186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7571c5e0-068c-4d67-b439-6ecfc8532a11 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:56.738511 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7571c5e0-068c-4d67-b439-6ecfc8532a11 admin admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:56.739287 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7571c5e0-068c-4d67-b439-6ecfc8532a11 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 16:22:56.739782 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7571c5e0-068c-4d67-b439-6ecfc8532a11 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:56.741550 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7571c5e0-068c-4d67-b439-6ecfc8532a11 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:56.756575 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1261/2503] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 16:22:56 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 16:22:56.761238 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7571c5e0-068c-4d67-b439-6ecfc8532a11 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:56.765949 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6cb203c-5066-4aca-9300-b8abe20d896d None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:56.771301 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43aa4b06-5267-4fa9-ad47-fb1a8998abf5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:56.771968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43aa4b06-5267-4fa9-ad47-fb1a8998abf5 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:56.772276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43aa4b06-5267-4fa9-ad47-fb1a8998abf5 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:56.772552 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43aa4b06-5267-4fa9-ad47-fb1a8998abf5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:56.774515 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7571c5e0-068c-4d67-b439-6ecfc8532a11 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:56.780371 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce0846f-097d-4645-a9a7-86c710e03242 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:56.781052 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce0846f-097d-4645-a9a7-86c710e03242 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:56.781438 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce0846f-097d-4645-a9a7-86c710e03242 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:56.781779 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce0846f-097d-4645-a9a7-86c710e03242 admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:56.782533 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce0846f-097d-4645-a9a7-86c710e03242 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:56.784415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43aa4b06-5267-4fa9-ad47-fb1a8998abf5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:56.784694 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce0846f-097d-4645-a9a7-86c710e03242 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:56.785858 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43aa4b06-5267-4fa9-ad47-fb1a8998abf5 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:56.803967 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1243/2504] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:22:56 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:22:56.819109 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1262/2505] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:22:56 2022] POST /identity/v3/projects => generated 374 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:22:56.825940 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43aa4b06-5267-4fa9-ad47-fb1a8998abf5 service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:56.836527 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43aa4b06-5267-4fa9-ad47-fb1a8998abf5 service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:56.899650 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000458fa-d220-438b-bd7e-90f856a15acf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:56.900267 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000458fa-d220-438b-bd7e-90f856a15acf admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:56.900585 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000458fa-d220-438b-bd7e-90f856a15acf admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:56.900867 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000458fa-d220-438b-bd7e-90f856a15acf admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:56.901504 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000458fa-d220-438b-bd7e-90f856a15acf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:56.903606 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000458fa-d220-438b-bd7e-90f856a15acf admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:56.930539 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1244/2506] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:22:56 2022] POST /identity/v3/users => generated 288 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:22:56.931595 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce0846f-097d-4645-a9a7-86c710e03242 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:56.942760 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce0846f-097d-4645-a9a7-86c710e03242 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:56.947711 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc58063-b0c8-4a63-afad-31ae1a1752e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:56.948397 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc58063-b0c8-4a63-afad-31ae1a1752e2 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:56.948754 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc58063-b0c8-4a63-afad-31ae1a1752e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:56.949101 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc58063-b0c8-4a63-afad-31ae1a1752e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:56.950553 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc58063-b0c8-4a63-afad-31ae1a1752e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:56.952434 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc58063-b0c8-4a63-afad-31ae1a1752e2 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:56.963874 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1263/2507] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:22:56 2022] GET /identity/v3/roles => generated 1427 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:22:56.968537 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000458fa-d220-438b-bd7e-90f856a15acf admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:56.979975 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000458fa-d220-438b-bd7e-90f856a15acf admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:56.986907 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a39167e-1ef5-4651-a0ab-9bfa6754dac4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:56.987607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a39167e-1ef5-4651-a0ab-9bfa6754dac4 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:56.988021 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a39167e-1ef5-4651-a0ab-9bfa6754dac4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:56.988370 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a39167e-1ef5-4651-a0ab-9bfa6754dac4 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:56.989102 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a39167e-1ef5-4651-a0ab-9bfa6754dac4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:56.990787 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a39167e-1ef5-4651-a0ab-9bfa6754dac4 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:57.006141 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1245/2508] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:22:56 2022] GET /identity/v3/roles => generated 1427 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:22:57.007315 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc58063-b0c8-4a63-afad-31ae1a1752e2 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:57.019994 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc58063-b0c8-4a63-afad-31ae1a1752e2 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:57.025256 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4895ce6-f3de-4a5e-a5e1-b4f03976d4e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:57.025885 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4895ce6-f3de-4a5e-a5e1-b4f03976d4e3 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:57.026185 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4895ce6-f3de-4a5e-a5e1-b4f03976d4e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:57.026493 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4895ce6-f3de-4a5e-a5e1-b4f03976d4e3 admin admin] PATH_INFO: `/v3/projects/f53ca6f714944d39b0be393a31581e68/users/94c85145df4f4c43a5f12f873d273543/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:57.043568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4895ce6-f3de-4a5e-a5e1-b4f03976d4e3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f53ca6f714944d39b0be393a31581e68, user_id=94c85145df4f4c43a5f12f873d273543, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:57.046375 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4895ce6-f3de-4a5e-a5e1-b4f03976d4e3 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:57.063439 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1264/2509] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:22:57 2022] PUT /identity/v3/projects/f53ca6f714944d39b0be393a31581e68/users/94c85145df4f4c43a5f12f873d273543/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:22:57.075358 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e873f51e-c2c8-4a16-8afe-0d59274e7752 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:57.075358 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e873f51e-c2c8-4a16-8afe-0d59274e7752 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:57.075586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e873f51e-c2c8-4a16-8afe-0d59274e7752 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:57.083381 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=94,keystone:DELETE=2,keystone:INSERT=4 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:57.110519 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-e873f51e-c2c8-4a16-8afe-0d59274e7752 None None] MFA Rules not processed for user `94c85145df4f4c43a5f12f873d273543`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:57.111979 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-e873f51e-c2c8-4a16-8afe-0d59274e7752 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:57.130334 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4895ce6-f3de-4a5e-a5e1-b4f03976d4e3 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:57.132278 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-d4895ce6-f3de-4a5e-a5e1-b4f03976d4e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:57.190032 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1246/2510] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:57 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 16:22:57.194056 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e873f51e-c2c8-4a16-8afe-0d59274e7752 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:57.220832 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4895ce6-f3de-4a5e-a5e1-b4f03976d4e3 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:57.225600 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf2cd51c-3edd-4c94-b839-07a5011632c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:57.226235 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2cd51c-3edd-4c94-b839-07a5011632c6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:57.226544 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2cd51c-3edd-4c94-b839-07a5011632c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:57.226821 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2cd51c-3edd-4c94-b839-07a5011632c6 admin admin] PATH_INFO: `/v3/projects/786884b7dee04cb4b508adc4a7e1aca0` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:57.227995 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf2cd51c-3edd-4c94-b839-07a5011632c6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=786884b7dee04cb4b508adc4a7e1aca0)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:57.229921 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf2cd51c-3edd-4c94-b839-07a5011632c6 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:57.263154 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e873f51e-c2c8-4a16-8afe-0d59274e7752 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:57.268398 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b384b1-9128-4d87-ba6d-b56ad36bee00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:57.269009 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b384b1-9128-4d87-ba6d-b56ad36bee00 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:57.269288 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b384b1-9128-4d87-ba6d-b56ad36bee00 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:57.269560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b384b1-9128-4d87-ba6d-b56ad36bee00 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:57.270454 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b384b1-9128-4d87-ba6d-b56ad36bee00 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:57.272478 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b384b1-9128-4d87-ba6d-b56ad36bee00 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:57.284375 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-bf2cd51c-3edd-4c94-b839-07a5011632c6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '786884b7dee04cb4b508adc4a7e1aca0', 'request_id': 'req-bf2cd51c-3edd-4c94-b839-07a5011632c6'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:57.293750 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-bf2cd51c-3edd-4c94-b839-07a5011632c6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '786884b7dee04cb4b508adc4a7e1aca0', 'request_id': 'req-bf2cd51c-3edd-4c94-b839-07a5011632c6'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:57.303151 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-bf2cd51c-3edd-4c94-b839-07a5011632c6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '786884b7dee04cb4b508adc4a7e1aca0', 'request_id': 'req-bf2cd51c-3edd-4c94-b839-07a5011632c6'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:57.304305 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1247/2511] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:22:57 2022] POST /identity/v3/projects => generated 374 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 16:22:57.308488 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b384b1-9128-4d87-ba6d-b56ad36bee00 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:57.319174 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-bf2cd51c-3edd-4c94-b839-07a5011632c6 admin admin] The token cache is being invalidate because project 786884b7dee04cb4b508adc4a7e1aca0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:22:57.319539 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-bf2cd51c-3edd-4c94-b839-07a5011632c6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:57.320430 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-bf2cd51c-3edd-4c94-b839-07a5011632c6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:57.322584 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1265/2512] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:22:57 2022] DELETE /identity/v3/projects/786884b7dee04cb4b508adc4a7e1aca0 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:22:57.328061 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539962a5-8a91-480e-81d7-7613925faafb None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:57.328349 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539962a5-8a91-480e-81d7-7613925faafb None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:57.328626 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539962a5-8a91-480e-81d7-7613925faafb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:57.344109 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:57.350568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-539962a5-8a91-480e-81d7-7613925faafb None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:57.352684 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-539962a5-8a91-480e-81d7-7613925faafb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:57.389851 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b384b1-9128-4d87-ba6d-b56ad36bee00 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:57.394580 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a172e8-5f2e-4371-be18-c6b137f7c1e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:57.395180 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a172e8-5f2e-4371-be18-c6b137f7c1e7 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:57.395461 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a172e8-5f2e-4371-be18-c6b137f7c1e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:57.395814 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a172e8-5f2e-4371-be18-c6b137f7c1e7 admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:57.396434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a172e8-5f2e-4371-be18-c6b137f7c1e7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:57.398351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a172e8-5f2e-4371-be18-c6b137f7c1e7 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:57.416953 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1266/2513] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:57 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:22:57.419505 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1248/2514] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:22:57 2022] POST /identity/v3/users => generated 294 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:22:57.423181 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539962a5-8a91-480e-81d7-7613925faafb None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:57.424592 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-539962a5-8a91-480e-81d7-7613925faafb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:57.426279 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5338fcd-4476-487a-a677-fc54e96022f6 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:57.426642 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5338fcd-4476-487a-a677-fc54e96022f6 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:57.426992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5338fcd-4476-487a-a677-fc54e96022f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:57.437202 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539962a5-8a91-480e-81d7-7613925faafb None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:57.441855 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8da161-8baf-44c5-801c-c5ed799b2a51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:57.442137 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:57.442457 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8da161-8baf-44c5-801c-c5ed799b2a51 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:57.442735 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8da161-8baf-44c5-801c-c5ed799b2a51 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:57.443166 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8da161-8baf-44c5-801c-c5ed799b2a51 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:57.444067 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d8da161-8baf-44c5-801c-c5ed799b2a51 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:57.446783 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d8da161-8baf-44c5-801c-c5ed799b2a51 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:57.449511 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-e5338fcd-4476-487a-a677-fc54e96022f6 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:57.450887 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-e5338fcd-4476-487a-a677-fc54e96022f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:57.458931 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1267/2515] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:22:57 2022] GET /identity/v3/roles => generated 1427 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:22:57.460104 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1249/2516] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:57 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 16:22:57.462952 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8da161-8baf-44c5-801c-c5ed799b2a51 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:57.464461 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5338fcd-4476-487a-a677-fc54e96022f6 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:57.473529 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8da161-8baf-44c5-801c-c5ed799b2a51 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:57.473728 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5338fcd-4476-487a-a677-fc54e96022f6 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:57.478313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c11b82-b26e-4598-8460-e39a43af63e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:57.478586 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5148e5-4e10-44eb-8a1d-748132ad943e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:57.478906 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c11b82-b26e-4598-8460-e39a43af63e2 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:57.479184 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c11b82-b26e-4598-8460-e39a43af63e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:57.479303 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5148e5-4e10-44eb-8a1d-748132ad943e admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:57.479452 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c11b82-b26e-4598-8460-e39a43af63e2 admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:57.479560 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5148e5-4e10-44eb-8a1d-748132ad943e admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:57.479796 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5148e5-4e10-44eb-8a1d-748132ad943e admin admin] PATH_INFO: `/v3/projects/c3c6acb87e03462da9ae56a61e65cafb/users/be56c66194cf43c3806b33a4ffaed685/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:57.480136 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c11b82-b26e-4598-8460-e39a43af63e2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 16:22:57.480962 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c11b82-b26e-4598-8460-e39a43af63e2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:57.482894 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c11b82-b26e-4598-8460-e39a43af63e2 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:57.494724 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1250/2517] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 16:22:57 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 16:22:57.495454 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5148e5-4e10-44eb-8a1d-748132ad943e admin admin] RBAC: Authorizing `identity:create_grant(project_id=c3c6acb87e03462da9ae56a61e65cafb, user_id=be56c66194cf43c3806b33a4ffaed685, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:57.498808 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5148e5-4e10-44eb-8a1d-748132ad943e admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:57.500540 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c11b82-b26e-4598-8460-e39a43af63e2 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:57.512082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c11b82-b26e-4598-8460-e39a43af63e2 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:57.512944 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1268/2518] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:22:57 2022] PUT /identity/v3/projects/c3c6acb87e03462da9ae56a61e65cafb/users/be56c66194cf43c3806b33a4ffaed685/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:22:57.517215 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5148e5-4e10-44eb-8a1d-748132ad943e admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:57.576178 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c234b962-96e9-41f3-841b-9dedeffc78a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:57.576700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c234b962-96e9-41f3-841b-9dedeffc78a4 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:57.576979 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c234b962-96e9-41f3-841b-9dedeffc78a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:57.577263 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c234b962-96e9-41f3-841b-9dedeffc78a4 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:57.577890 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c234b962-96e9-41f3-841b-9dedeffc78a4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:57.579840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c234b962-96e9-41f3-841b-9dedeffc78a4 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:57.587821 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5148e5-4e10-44eb-8a1d-748132ad943e admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:57.593516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24280fb8-6e5a-45dd-9809-96e326cce0b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:57.594119 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24280fb8-6e5a-45dd-9809-96e326cce0b2 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:57.594423 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24280fb8-6e5a-45dd-9809-96e326cce0b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:57.594698 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24280fb8-6e5a-45dd-9809-96e326cce0b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:57.595345 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24280fb8-6e5a-45dd-9809-96e326cce0b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:57.596976 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24280fb8-6e5a-45dd-9809-96e326cce0b2 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:57.608636 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1251/2519] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:22:57 2022] POST /identity/v3/projects => generated 358 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 16:22:57.611682 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1269/2520] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:22:57 2022] GET /identity/v3/roles => generated 1427 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 16:22:57.613287 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c234b962-96e9-41f3-841b-9dedeffc78a4 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:57.613991 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24280fb8-6e5a-45dd-9809-96e326cce0b2 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:57.623819 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24280fb8-6e5a-45dd-9809-96e326cce0b2 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:57.676360 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c234b962-96e9-41f3-841b-9dedeffc78a4 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:57.681428 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba69782-8751-49f1-990b-ab2a7b71b345 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:57.682026 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba69782-8751-49f1-990b-ab2a7b71b345 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:57.682302 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba69782-8751-49f1-990b-ab2a7b71b345 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:57.682568 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba69782-8751-49f1-990b-ab2a7b71b345 admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:57.683112 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a5f9c5-4e83-4f66-b597-78147812640e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:57.683244 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ba69782-8751-49f1-990b-ab2a7b71b345 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:57.683834 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a5f9c5-4e83-4f66-b597-78147812640e admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:57.684119 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a5f9c5-4e83-4f66-b597-78147812640e admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:57.684502 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a5f9c5-4e83-4f66-b597-78147812640e admin admin] PATH_INFO: `/v3/projects/c3c6acb87e03462da9ae56a61e65cafb/users/be56c66194cf43c3806b33a4ffaed685/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:57.685313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ba69782-8751-49f1-990b-ab2a7b71b345 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:57.686592 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a5f9c5-4e83-4f66-b597-78147812640e admin admin] RBAC: Authorizing `identity:create_grant(project_id=c3c6acb87e03462da9ae56a61e65cafb, user_id=be56c66194cf43c3806b33a4ffaed685, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:57.689349 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a5f9c5-4e83-4f66-b597-78147812640e admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:57.700575 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1270/2521] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:22:57 2022] PUT /identity/v3/projects/c3c6acb87e03462da9ae56a61e65cafb/users/be56c66194cf43c3806b33a4ffaed685/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 88 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:22:57.711945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5dc167-5989-4167-b5bd-331bf84f8149 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:57.712256 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5dc167-5989-4167-b5bd-331bf84f8149 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:57.712535 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5dc167-5989-4167-b5bd-331bf84f8149 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:57.716415 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1252/2522] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:22:57 2022] POST /identity/v3/users => generated 280 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:22:57.717784 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba69782-8751-49f1-990b-ab2a7b71b345 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:57.720606 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:57.729429 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba69782-8751-49f1-990b-ab2a7b71b345 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:57.744741 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-ae5dc167-5989-4167-b5bd-331bf84f8149 None None] MFA Rules not processed for user `be56c66194cf43c3806b33a4ffaed685`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:57.745986 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-ae5dc167-5989-4167-b5bd-331bf84f8149 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:57.786165 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a840b1-b490-40f2-86de-8103f9be624c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:57.786768 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a840b1-b490-40f2-86de-8103f9be624c admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:57.787052 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a840b1-b490-40f2-86de-8103f9be624c admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:57.787322 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a840b1-b490-40f2-86de-8103f9be624c admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:57.788011 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a840b1-b490-40f2-86de-8103f9be624c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:57.789604 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a840b1-b490-40f2-86de-8103f9be624c admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:57.791698 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:57.805978 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1253/2523] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:22:57 2022] GET /identity/v3/roles => generated 1427 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:22:57.807172 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a840b1-b490-40f2-86de-8103f9be624c admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:57.818458 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a840b1-b490-40f2-86de-8103f9be624c admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:57.820181 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1271/2524] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:57 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 16:22:57.823830 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3db0da-caab-4027-a010-c2d422b6005f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:57.825277 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3db0da-caab-4027-a010-c2d422b6005f admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:57.825563 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3db0da-caab-4027-a010-c2d422b6005f admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:57.825840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3db0da-caab-4027-a010-c2d422b6005f admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:57.826561 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3db0da-caab-4027-a010-c2d422b6005f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:57.828171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3db0da-caab-4027-a010-c2d422b6005f admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:57.838478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5dc167-5989-4167-b5bd-331bf84f8149 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:57.841727 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-ae5dc167-5989-4167-b5bd-331bf84f8149 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:57.848149 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1254/2525] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:22:57 2022] GET /identity/v3/roles => generated 1427 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:22:57.850316 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3db0da-caab-4027-a010-c2d422b6005f admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:57.862968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3db0da-caab-4027-a010-c2d422b6005f admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:57.868466 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f82790-d098-4aa7-ae44-155e35d57792 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:57.869165 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f82790-d098-4aa7-ae44-155e35d57792 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:57.869450 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f82790-d098-4aa7-ae44-155e35d57792 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:57.869728 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f82790-d098-4aa7-ae44-155e35d57792 admin admin] PATH_INFO: `/v3/projects/c99de09fe71740b0b110d39a97d35768/users/788df54c872945adb331ebc045a82c42/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:57.888867 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f82790-d098-4aa7-ae44-155e35d57792 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c99de09fe71740b0b110d39a97d35768, user_id=788df54c872945adb331ebc045a82c42, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:57.891594 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f82790-d098-4aa7-ae44-155e35d57792 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:57.908555 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1255/2526] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:22:57 2022] PUT /identity/v3/projects/c99de09fe71740b0b110d39a97d35768/users/788df54c872945adb331ebc045a82c42/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:22:57.919295 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb5a710-4a44-4bef-aec4-fcdc144b8774 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:57.919600 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb5a710-4a44-4bef-aec4-fcdc144b8774 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:57.920256 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb5a710-4a44-4bef-aec4-fcdc144b8774 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:57.929988 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5dc167-5989-4167-b5bd-331bf84f8149 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:57.953626 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-cfb5a710-4a44-4bef-aec4-fcdc144b8774 None None] MFA Rules not processed for user `788df54c872945adb331ebc045a82c42`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:57.954897 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-cfb5a710-4a44-4bef-aec4-fcdc144b8774 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:58.009334 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92569963-5f8a-464e-bb6a-f1add4ff06d6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:58.009947 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92569963-5f8a-464e-bb6a-f1add4ff06d6 service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:58.010228 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92569963-5f8a-464e-bb6a-f1add4ff06d6 service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:58.010509 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92569963-5f8a-464e-bb6a-f1add4ff06d6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:58.036607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1256/2527] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:57 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 16:22:58.073810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:58.080044 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92569963-5f8a-464e-bb6a-f1add4ff06d6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:58.081931 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92569963-5f8a-464e-bb6a-f1add4ff06d6 service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:58.100568 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1272/2528] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:22:57 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 259 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 16:22:58.171997 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb5a710-4a44-4bef-aec4-fcdc144b8774 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:58.173906 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-cfb5a710-4a44-4bef-aec4-fcdc144b8774 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:58.181631 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92569963-5f8a-464e-bb6a-f1add4ff06d6 service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:58.191499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92569963-5f8a-464e-bb6a-f1add4ff06d6 service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:58.192578 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb5a710-4a44-4bef-aec4-fcdc144b8774 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:58.196334 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25374115-c3c4-447d-a80f-89e31f54aa1a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:58.196740 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60c532e-2a15-4884-8c95-d5dacc5b5bb5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:58.196941 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25374115-c3c4-447d-a80f-89e31f54aa1a service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:58.197215 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25374115-c3c4-447d-a80f-89e31f54aa1a service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:58.197514 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60c532e-2a15-4884-8c95-d5dacc5b5bb5 service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:58.197617 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25374115-c3c4-447d-a80f-89e31f54aa1a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:58.197802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60c532e-2a15-4884-8c95-d5dacc5b5bb5 service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:58.198081 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60c532e-2a15-4884-8c95-d5dacc5b5bb5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:58.200435 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-a60c532e-2a15-4884-8c95-d5dacc5b5bb5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:58.216972 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25374115-c3c4-447d-a80f-89e31f54aa1a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:58.218701 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25374115-c3c4-447d-a80f-89e31f54aa1a service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:58.230161 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1257/2529] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:22:58 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:22:58.250348 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25374115-c3c4-447d-a80f-89e31f54aa1a service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:58.291283 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60c532e-2a15-4884-8c95-d5dacc5b5bb5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:58.293836 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60c532e-2a15-4884-8c95-d5dacc5b5bb5 service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:58.305598 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1273/2530] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:22:58 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:22:58.327257 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25374115-c3c4-447d-a80f-89e31f54aa1a service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:58.332818 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2189dba-85ed-4ef0-9032-0719bc3474aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:58.333465 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2189dba-85ed-4ef0-9032-0719bc3474aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:58.333749 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2189dba-85ed-4ef0-9032-0719bc3474aa admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:58.334023 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2189dba-85ed-4ef0-9032-0719bc3474aa admin admin] PATH_INFO: `/v3/users/788df54c872945adb331ebc045a82c42` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:58.335208 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2189dba-85ed-4ef0-9032-0719bc3474aa admin admin] RBAC: Authorizing `identity:delete_user(user_id=788df54c872945adb331ebc045a82c42)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:58.337166 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2189dba-85ed-4ef0-9032-0719bc3474aa admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:58.388549 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=93,keystone:INSERT=1,keystone:DELETE=6 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:58.411219 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-b2189dba-85ed-4ef0-9032-0719bc3474aa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '788df54c872945adb331ebc045a82c42'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:58.412391 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-b2189dba-85ed-4ef0-9032-0719bc3474aa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '788df54c872945adb331ebc045a82c42'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:58.421276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-b2189dba-85ed-4ef0-9032-0719bc3474aa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '788df54c872945adb331ebc045a82c42'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:58.436144 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-b2189dba-85ed-4ef0-9032-0719bc3474aa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '788df54c872945adb331ebc045a82c42'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:58.436906 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-b2189dba-85ed-4ef0-9032-0719bc3474aa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '788df54c872945adb331ebc045a82c42'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:58.457881 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1258/2531] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:22:58 2022] DELETE /identity/v3/users/788df54c872945adb331ebc045a82c42 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:22:58.462529 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e772b0-48e6-4f46-a43b-f560c10063e8 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:58.462746 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e772b0-48e6-4f46-a43b-f560c10063e8 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:58.463025 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e772b0-48e6-4f46-a43b-f560c10063e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:58.484939 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-57e772b0-48e6-4f46-a43b-f560c10063e8 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:58.486549 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-57e772b0-48e6-4f46-a43b-f560c10063e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:58.556308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1274/2532] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:58 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:22:58.568713 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2189dba-85ed-4ef0-9032-0719bc3474aa admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:58.570835 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-b2189dba-85ed-4ef0-9032-0719bc3474aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:58.629425 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2189dba-85ed-4ef0-9032-0719bc3474aa admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:58.633092 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4150072-f3cb-4747-afa7-b7a431afd61f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:58.633681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4150072-f3cb-4747-afa7-b7a431afd61f service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:58.633958 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4150072-f3cb-4747-afa7-b7a431afd61f service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:58.634224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4150072-f3cb-4747-afa7-b7a431afd61f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:58.643940 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4150072-f3cb-4747-afa7-b7a431afd61f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:58.645352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4150072-f3cb-4747-afa7-b7a431afd61f service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:58.654907 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1259/2533] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:22:58 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 16:22:58.930529 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e772b0-48e6-4f46-a43b-f560c10063e8 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:58.932105 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-57e772b0-48e6-4f46-a43b-f560c10063e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:58.946332 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e772b0-48e6-4f46-a43b-f560c10063e8 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:58.951505 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a66cdbc-3102-41ac-a109-d08dd7fb6ac3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:58.952191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a66cdbc-3102-41ac-a109-d08dd7fb6ac3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:58.952473 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a66cdbc-3102-41ac-a109-d08dd7fb6ac3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:58.952771 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a66cdbc-3102-41ac-a109-d08dd7fb6ac3 admin admin] PATH_INFO: `/v3/projects/c99de09fe71740b0b110d39a97d35768` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:58.953920 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a66cdbc-3102-41ac-a109-d08dd7fb6ac3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c99de09fe71740b0b110d39a97d35768)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:58.956508 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a66cdbc-3102-41ac-a109-d08dd7fb6ac3 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:59.004694 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-2a66cdbc-3102-41ac-a109-d08dd7fb6ac3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c99de09fe71740b0b110d39a97d35768', 'request_id': 'req-2a66cdbc-3102-41ac-a109-d08dd7fb6ac3'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:59.010936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-2a66cdbc-3102-41ac-a109-d08dd7fb6ac3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c99de09fe71740b0b110d39a97d35768', 'request_id': 'req-2a66cdbc-3102-41ac-a109-d08dd7fb6ac3'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:59.016984 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-2a66cdbc-3102-41ac-a109-d08dd7fb6ac3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c99de09fe71740b0b110d39a97d35768', 'request_id': 'req-2a66cdbc-3102-41ac-a109-d08dd7fb6ac3'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:59.033768 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-2a66cdbc-3102-41ac-a109-d08dd7fb6ac3 admin admin] The token cache is being invalidate because project c99de09fe71740b0b110d39a97d35768 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:22:59.034097 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-2a66cdbc-3102-41ac-a109-d08dd7fb6ac3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:59.034832 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-2a66cdbc-3102-41ac-a109-d08dd7fb6ac3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:59.036500 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1275/2534] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:22:58 2022] DELETE /identity/v3/projects/c99de09fe71740b0b110d39a97d35768 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:22:59.043749 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042b4d5c-fd1a-4da3-a6be-534d3c45c26b None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:59.044048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042b4d5c-fd1a-4da3-a6be-534d3c45c26b None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:59.044291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042b4d5c-fd1a-4da3-a6be-534d3c45c26b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:59.064062 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-042b4d5c-fd1a-4da3-a6be-534d3c45c26b None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:59.065259 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-042b4d5c-fd1a-4da3-a6be-534d3c45c26b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:59.125986 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1260/2535] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:59 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:22:59.133930 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3483e80e-a7cd-4327-ba4f-0985997ba0b6 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:59.134216 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3483e80e-a7cd-4327-ba4f-0985997ba0b6 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:59.134486 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3483e80e-a7cd-4327-ba4f-0985997ba0b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:59.154590 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-3483e80e-a7cd-4327-ba4f-0985997ba0b6 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:59.155817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-3483e80e-a7cd-4327-ba4f-0985997ba0b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:59.163673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1276/2536] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:59 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:22:59.168705 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042b4d5c-fd1a-4da3-a6be-534d3c45c26b None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:59.179116 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042b4d5c-fd1a-4da3-a6be-534d3c45c26b None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:59.183868 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8520ae-9c64-4287-8014-88d29b7beebb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:59.184478 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8520ae-9c64-4287-8014-88d29b7beebb admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:59.184760 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8520ae-9c64-4287-8014-88d29b7beebb admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:59.185031 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8520ae-9c64-4287-8014-88d29b7beebb admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:59.185691 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8520ae-9c64-4287-8014-88d29b7beebb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 16:22:59.185996 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8520ae-9c64-4287-8014-88d29b7beebb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:59.187614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8520ae-9c64-4287-8014-88d29b7beebb admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:59.200727 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1261/2537] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 16:22:59 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 16:22:59.204967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3483e80e-a7cd-4327-ba4f-0985997ba0b6 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:59.215748 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3483e80e-a7cd-4327-ba4f-0985997ba0b6 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:59.220998 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54376800-72bb-4e02-9113-9213d8746707 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:59.221622 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54376800-72bb-4e02-9113-9213d8746707 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:59.221910 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54376800-72bb-4e02-9113-9213d8746707 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:59.222191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54376800-72bb-4e02-9113-9213d8746707 admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:59.222845 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54376800-72bb-4e02-9113-9213d8746707 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:59.224807 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54376800-72bb-4e02-9113-9213d8746707 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:59.231231 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:59.259709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1277/2538] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:22:59 2022] POST /identity/v3/projects => generated 380 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:22:59.262394 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8520ae-9c64-4287-8014-88d29b7beebb admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:59.275970 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8520ae-9c64-4287-8014-88d29b7beebb admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:59.327042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:59.345173 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e4c26b-5fa1-40c9-ad1e-5308c051fdf6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:59.345331 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54376800-72bb-4e02-9113-9213d8746707 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:59.345793 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e4c26b-5fa1-40c9-ad1e-5308c051fdf6 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:59.346082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e4c26b-5fa1-40c9-ad1e-5308c051fdf6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:59.346351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e4c26b-5fa1-40c9-ad1e-5308c051fdf6 admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:59.347028 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-54376800-72bb-4e02-9113-9213d8746707 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:59.347126 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e4c26b-5fa1-40c9-ad1e-5308c051fdf6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:59.349024 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e4c26b-5fa1-40c9-ad1e-5308c051fdf6 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:59.360794 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54376800-72bb-4e02-9113-9213d8746707 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:59.365784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78bd877f-1187-4788-ac40-e1349099b99e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:59.366415 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bd877f-1187-4788-ac40-e1349099b99e admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:59.366699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bd877f-1187-4788-ac40-e1349099b99e admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:59.366983 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bd877f-1187-4788-ac40-e1349099b99e admin admin] PATH_INFO: `/v3/users/93b8d0c807664c4488400401efcee801` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:59.368179 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78bd877f-1187-4788-ac40-e1349099b99e admin admin] RBAC: Authorizing `identity:delete_user(user_id=93b8d0c807664c4488400401efcee801)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:59.370123 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78bd877f-1187-4788-ac40-e1349099b99e admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:59.372079 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1262/2539] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:22:59 2022] POST /identity/v3/users => generated 291 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:22:59.376270 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e4c26b-5fa1-40c9-ad1e-5308c051fdf6 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:59.387432 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e4c26b-5fa1-40c9-ad1e-5308c051fdf6 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:59.392555 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a956ca11-0c5a-48ee-ae9f-43ba182ff21d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:59.393210 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a956ca11-0c5a-48ee-ae9f-43ba182ff21d admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:59.393492 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a956ca11-0c5a-48ee-ae9f-43ba182ff21d admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:59.393788 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a956ca11-0c5a-48ee-ae9f-43ba182ff21d admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:59.394432 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a956ca11-0c5a-48ee-ae9f-43ba182ff21d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:59.396067 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a956ca11-0c5a-48ee-ae9f-43ba182ff21d admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:59.407538 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1263/2540] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:22:59 2022] GET /identity/v3/roles => generated 1427 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:22:59.411748 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a956ca11-0c5a-48ee-ae9f-43ba182ff21d admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:59.423202 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a956ca11-0c5a-48ee-ae9f-43ba182ff21d admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:59.427257 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-78bd877f-1187-4788-ac40-e1349099b99e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '93b8d0c807664c4488400401efcee801'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:59.427921 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-78bd877f-1187-4788-ac40-e1349099b99e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '93b8d0c807664c4488400401efcee801'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:59.428192 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e74a39-5c65-4386-be9f-675224f47430 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:59.428795 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-78bd877f-1187-4788-ac40-e1349099b99e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '93b8d0c807664c4488400401efcee801'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:59.428918 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e74a39-5c65-4386-be9f-675224f47430 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:59.429088 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e74a39-5c65-4386-be9f-675224f47430 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:59.429334 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e74a39-5c65-4386-be9f-675224f47430 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:59.429982 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e74a39-5c65-4386-be9f-675224f47430 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:59.432326 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e74a39-5c65-4386-be9f-675224f47430 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:59.438279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-78bd877f-1187-4788-ac40-e1349099b99e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '93b8d0c807664c4488400401efcee801'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:59.445069 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1264/2541] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:22:59 2022] GET /identity/v3/roles => generated 1427 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 16:22:59.448768 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e74a39-5c65-4386-be9f-675224f47430 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:59.452169 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-c5e74a39-5c65-4386-be9f-675224f47430 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:59.453666 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-78bd877f-1187-4788-ac40-e1349099b99e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '93b8d0c807664c4488400401efcee801'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:22:59.472768 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e74a39-5c65-4386-be9f-675224f47430 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:59.473960 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1278/2542] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:22:59 2022] DELETE /identity/v3/users/93b8d0c807664c4488400401efcee801 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:22:59.478546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbf8bfe-85cc-4e40-8d10-d80290e930f8 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:59.478835 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbf8bfe-85cc-4e40-8d10-d80290e930f8 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:59.479112 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbf8bfe-85cc-4e40-8d10-d80290e930f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:59.499894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-abbf8bfe-85cc-4e40-8d10-d80290e930f8 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:59.501107 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-abbf8bfe-85cc-4e40-8d10-d80290e930f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:59.538743 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4deca1a3-114f-4fec-bb47-31810ad226c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:59.539639 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4deca1a3-114f-4fec-bb47-31810ad226c5 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:59.539929 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4deca1a3-114f-4fec-bb47-31810ad226c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:59.540268 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4deca1a3-114f-4fec-bb47-31810ad226c5 admin admin] PATH_INFO: `/v3/projects/1259ac84ae764120b41de4af4b0cdb56/users/fe64f798929444dd838825dc0d2700cf/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:59.557577 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4deca1a3-114f-4fec-bb47-31810ad226c5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1259ac84ae764120b41de4af4b0cdb56, user_id=fe64f798929444dd838825dc0d2700cf, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:59.560900 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4deca1a3-114f-4fec-bb47-31810ad226c5 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:59.577195 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1279/2543] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:59 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 16:22:59.583766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1265/2544] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:22:59 2022] PUT /identity/v3/projects/1259ac84ae764120b41de4af4b0cdb56/users/fe64f798929444dd838825dc0d2700cf/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:22:59.587539 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbf8bfe-85cc-4e40-8d10-d80290e930f8 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:59.588268 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f9f938-abdf-4b7a-b067-ea757bee4641 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:59.588575 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f9f938-abdf-4b7a-b067-ea757bee4641 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:59.589855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f9f938-abdf-4b7a-b067-ea757bee4641 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:59.590611 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-abbf8bfe-85cc-4e40-8d10-d80290e930f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:59.612784 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-31f9f938-abdf-4b7a-b067-ea757bee4641 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:22:59.614009 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-31f9f938-abdf-4b7a-b067-ea757bee4641 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:22:59.660430 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbf8bfe-85cc-4e40-8d10-d80290e930f8 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:59.678837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:59.695125 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1266/2545] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:22:59 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:22:59.707249 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f9f938-abdf-4b7a-b067-ea757bee4641 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:22:59.733154 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec14a2bc-d942-4ce5-bed3-f8850b924666 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:59.734185 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec14a2bc-d942-4ce5-bed3-f8850b924666 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:59.734496 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec14a2bc-d942-4ce5-bed3-f8850b924666 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:59.734846 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec14a2bc-d942-4ce5-bed3-f8850b924666 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:59.745064 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec14a2bc-d942-4ce5-bed3-f8850b924666 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:59.747184 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec14a2bc-d942-4ce5-bed3-f8850b924666 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:59.757647 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1280/2546] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:22:59 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 171 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:22:59.761909 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:22:59.782020 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f9f938-abdf-4b7a-b067-ea757bee4641 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:22:59.787299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e104dac8-a755-48f8-97ab-482d5e715d54 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:22:59.787999 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e104dac8-a755-48f8-97ab-482d5e715d54 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:22:59.788671 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e104dac8-a755-48f8-97ab-482d5e715d54 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:22:59.788979 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e104dac8-a755-48f8-97ab-482d5e715d54 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:22:59.804435 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e104dac8-a755-48f8-97ab-482d5e715d54 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:22:59.808229 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e104dac8-a755-48f8-97ab-482d5e715d54 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:22:59.828034 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1267/2547] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:22:59 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:23:00.142828 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec14a2bc-d942-4ce5-bed3-f8850b924666 service neutron] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:00.144561 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-ec14a2bc-d942-4ce5-bed3-f8850b924666 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:00.159445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec14a2bc-d942-4ce5-bed3-f8850b924666 service neutron] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:00.164662 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e19256-ac6c-4d1f-b51b-2cc50bbd7650 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:00.165326 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e19256-ac6c-4d1f-b51b-2cc50bbd7650 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:00.165605 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e19256-ac6c-4d1f-b51b-2cc50bbd7650 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:00.165876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e19256-ac6c-4d1f-b51b-2cc50bbd7650 admin admin] PATH_INFO: `/v3/projects/5d0f97e2d1654e9a899c645151123520` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:00.167182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e19256-ac6c-4d1f-b51b-2cc50bbd7650 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5d0f97e2d1654e9a899c645151123520)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:00.169823 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e19256-ac6c-4d1f-b51b-2cc50bbd7650 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:00.245062 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-c4e19256-ac6c-4d1f-b51b-2cc50bbd7650 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5d0f97e2d1654e9a899c645151123520', 'request_id': 'req-c4e19256-ac6c-4d1f-b51b-2cc50bbd7650'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:00.256307 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-c4e19256-ac6c-4d1f-b51b-2cc50bbd7650 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5d0f97e2d1654e9a899c645151123520', 'request_id': 'req-c4e19256-ac6c-4d1f-b51b-2cc50bbd7650'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:00.266583 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-c4e19256-ac6c-4d1f-b51b-2cc50bbd7650 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5d0f97e2d1654e9a899c645151123520', 'request_id': 'req-c4e19256-ac6c-4d1f-b51b-2cc50bbd7650'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:00.289993 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-c4e19256-ac6c-4d1f-b51b-2cc50bbd7650 admin admin] The token cache is being invalidate because project 5d0f97e2d1654e9a899c645151123520 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:23:00.290610 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-c4e19256-ac6c-4d1f-b51b-2cc50bbd7650 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:00.291576 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-c4e19256-ac6c-4d1f-b51b-2cc50bbd7650 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:00.294337 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1281/2548] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:23:00 2022] DELETE /identity/v3/projects/5d0f97e2d1654e9a899c645151123520 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:00.299096 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e104dac8-a755-48f8-97ab-482d5e715d54 service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:00.312334 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e104dac8-a755-48f8-97ab-482d5e715d54 service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:00.382798 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66ae096-7ba8-421c-8204-5afcb1cab8b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:00.383535 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66ae096-7ba8-421c-8204-5afcb1cab8b9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:00.383938 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66ae096-7ba8-421c-8204-5afcb1cab8b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:00.384280 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66ae096-7ba8-421c-8204-5afcb1cab8b9 admin admin] PATH_INFO: `/v3/users/feffac2c641d44f08156cc053039c7c8` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:00.385703 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66ae096-7ba8-421c-8204-5afcb1cab8b9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=feffac2c641d44f08156cc053039c7c8)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:00.388493 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66ae096-7ba8-421c-8204-5afcb1cab8b9 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:00.445397 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-f66ae096-7ba8-421c-8204-5afcb1cab8b9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'feffac2c641d44f08156cc053039c7c8'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:00.445976 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-f66ae096-7ba8-421c-8204-5afcb1cab8b9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'feffac2c641d44f08156cc053039c7c8'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:00.453951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-f66ae096-7ba8-421c-8204-5afcb1cab8b9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'feffac2c641d44f08156cc053039c7c8'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:00.471334 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-f66ae096-7ba8-421c-8204-5afcb1cab8b9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'feffac2c641d44f08156cc053039c7c8'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:00.472342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-f66ae096-7ba8-421c-8204-5afcb1cab8b9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'feffac2c641d44f08156cc053039c7c8'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:00.492538 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1268/2549] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:23:00 2022] DELETE /identity/v3/users/feffac2c641d44f08156cc053039c7c8 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:00.941797 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e19256-ac6c-4d1f-b51b-2cc50bbd7650 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:00.943518 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-c4e19256-ac6c-4d1f-b51b-2cc50bbd7650 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:00.964971 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e19256-ac6c-4d1f-b51b-2cc50bbd7650 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:01.069032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075a1bc8-be30-4723-a57f-94bb9f24d12c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:01.070093 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075a1bc8-be30-4723-a57f-94bb9f24d12c admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:01.070479 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075a1bc8-be30-4723-a57f-94bb9f24d12c admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:01.071092 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075a1bc8-be30-4723-a57f-94bb9f24d12c admin admin] PATH_INFO: `/v3/projects/9efc7f1cd12648ad87b86764887bde67` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:01.073409 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075a1bc8-be30-4723-a57f-94bb9f24d12c admin admin] RBAC: Authorizing `identity:delete_project(project_id=9efc7f1cd12648ad87b86764887bde67)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:01.080718 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075a1bc8-be30-4723-a57f-94bb9f24d12c admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:01.148582 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:01.155741 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-075a1bc8-be30-4723-a57f-94bb9f24d12c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9efc7f1cd12648ad87b86764887bde67', 'request_id': 'req-075a1bc8-be30-4723-a57f-94bb9f24d12c'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:01.166422 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-075a1bc8-be30-4723-a57f-94bb9f24d12c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9efc7f1cd12648ad87b86764887bde67', 'request_id': 'req-075a1bc8-be30-4723-a57f-94bb9f24d12c'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:01.176231 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-075a1bc8-be30-4723-a57f-94bb9f24d12c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9efc7f1cd12648ad87b86764887bde67', 'request_id': 'req-075a1bc8-be30-4723-a57f-94bb9f24d12c'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:01.198604 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-075a1bc8-be30-4723-a57f-94bb9f24d12c admin admin] The token cache is being invalidate because project 9efc7f1cd12648ad87b86764887bde67 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:23:01.199659 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-075a1bc8-be30-4723-a57f-94bb9f24d12c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:01.201256 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-075a1bc8-be30-4723-a57f-94bb9f24d12c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:01.206100 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1282/2550] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:23:00 2022] DELETE /identity/v3/projects/9efc7f1cd12648ad87b86764887bde67 => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:01.212458 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cdfaf1-203f-456c-b3db-03517fdfbeaf None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:01.214554 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cdfaf1-203f-456c-b3db-03517fdfbeaf None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:01.215672 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cdfaf1-203f-456c-b3db-03517fdfbeaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:01.246053 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-55cdfaf1-203f-456c-b3db-03517fdfbeaf None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:01.248567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-55cdfaf1-203f-456c-b3db-03517fdfbeaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:01.336131 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1269/2551] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:01 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:23:01.347621 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d00a46a-c39b-451f-8b0f-70200f4b4ec2 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:01.347942 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d00a46a-c39b-451f-8b0f-70200f4b4ec2 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:01.348225 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d00a46a-c39b-451f-8b0f-70200f4b4ec2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:01.386329 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-8d00a46a-c39b-451f-8b0f-70200f4b4ec2 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:01.388780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-8d00a46a-c39b-451f-8b0f-70200f4b4ec2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:01.404249 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1283/2552] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:01 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:23:01.405010 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55cdfaf1-203f-456c-b3db-03517fdfbeaf None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:01.410589 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:01.416840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55cdfaf1-203f-456c-b3db-03517fdfbeaf None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:01.426665 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e33c75-f083-47ab-961d-9cbe5b1ead14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:01.427444 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e33c75-f083-47ab-961d-9cbe5b1ead14 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:01.428012 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e33c75-f083-47ab-961d-9cbe5b1ead14 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:01.428775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e33c75-f083-47ab-961d-9cbe5b1ead14 admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:01.429798 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e33c75-f083-47ab-961d-9cbe5b1ead14 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 16:23:01.430326 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e33c75-f083-47ab-961d-9cbe5b1ead14 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:01.433182 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e33c75-f083-47ab-961d-9cbe5b1ead14 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:01.459361 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1270/2553] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 16:23:01 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 16:23:01.461498 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d00a46a-c39b-451f-8b0f-70200f4b4ec2 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:01.471582 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d00a46a-c39b-451f-8b0f-70200f4b4ec2 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:01.476483 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47c3dc2-142c-46ed-92cc-b3de69b0be77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:01.477035 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47c3dc2-142c-46ed-92cc-b3de69b0be77 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:01.477310 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47c3dc2-142c-46ed-92cc-b3de69b0be77 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:01.477586 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47c3dc2-142c-46ed-92cc-b3de69b0be77 admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:01.478826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c47c3dc2-142c-46ed-92cc-b3de69b0be77 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:01.480284 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c47c3dc2-142c-46ed-92cc-b3de69b0be77 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:01.519726 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1284/2554] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:23:01 2022] POST /identity/v3/projects => generated 388 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:23:01.520885 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e33c75-f083-47ab-961d-9cbe5b1ead14 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:01.536151 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e33c75-f083-47ab-961d-9cbe5b1ead14 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:01.632640 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe29e14-fa03-448f-b69e-2ef70dcc8f1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:01.633288 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe29e14-fa03-448f-b69e-2ef70dcc8f1c admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:01.633608 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe29e14-fa03-448f-b69e-2ef70dcc8f1c admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:01.633877 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe29e14-fa03-448f-b69e-2ef70dcc8f1c admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:01.634754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe29e14-fa03-448f-b69e-2ef70dcc8f1c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:01.637876 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe29e14-fa03-448f-b69e-2ef70dcc8f1c admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:01.666270 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1271/2555] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:23:01 2022] POST /identity/v3/users => generated 295 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:23:01.671872 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47c3dc2-142c-46ed-92cc-b3de69b0be77 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:01.687785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47c3dc2-142c-46ed-92cc-b3de69b0be77 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:01.694124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb5037f-1421-49d0-8562-f628c9620cd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:01.694889 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb5037f-1421-49d0-8562-f628c9620cd2 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:01.695243 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb5037f-1421-49d0-8562-f628c9620cd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:01.695592 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb5037f-1421-49d0-8562-f628c9620cd2 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:01.696498 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb5037f-1421-49d0-8562-f628c9620cd2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:01.698140 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb5037f-1421-49d0-8562-f628c9620cd2 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:01.713319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1285/2556] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:23:01 2022] GET /identity/v3/roles => generated 1427 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:23:01.716462 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe29e14-fa03-448f-b69e-2ef70dcc8f1c admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:01.728130 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe29e14-fa03-448f-b69e-2ef70dcc8f1c admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:01.734764 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002a1835-dee4-46d6-9c73-22665849712a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:01.735550 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002a1835-dee4-46d6-9c73-22665849712a admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:01.735906 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002a1835-dee4-46d6-9c73-22665849712a admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:01.736269 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002a1835-dee4-46d6-9c73-22665849712a admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:01.736997 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-002a1835-dee4-46d6-9c73-22665849712a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:01.738679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-002a1835-dee4-46d6-9c73-22665849712a admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:01.754474 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1272/2557] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:23:01 2022] GET /identity/v3/roles => generated 1427 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 16:23:01.757510 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb5037f-1421-49d0-8562-f628c9620cd2 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:01.770453 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb5037f-1421-49d0-8562-f628c9620cd2 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:01.779234 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bde4ca0-4e0a-4bca-b641-b310c5931929 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:01.779953 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bde4ca0-4e0a-4bca-b641-b310c5931929 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:01.780241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bde4ca0-4e0a-4bca-b641-b310c5931929 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:01.780578 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bde4ca0-4e0a-4bca-b641-b310c5931929 admin admin] PATH_INFO: `/v3/projects/6bd76ba8381947eebda2630779ef3b8a/users/9b247fad73ea4f34af0db811be6c0048/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:01.820831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bde4ca0-4e0a-4bca-b641-b310c5931929 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6bd76ba8381947eebda2630779ef3b8a, user_id=9b247fad73ea4f34af0db811be6c0048, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:01.824191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bde4ca0-4e0a-4bca-b641-b310c5931929 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:01.851815 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1286/2558] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:23:01 2022] PUT /identity/v3/projects/6bd76ba8381947eebda2630779ef3b8a/users/9b247fad73ea4f34af0db811be6c0048/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:01.859222 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ded99f1-3b43-4e81-80fa-21ee12b18e23 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:01.859508 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ded99f1-3b43-4e81-80fa-21ee12b18e23 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:01.859935 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ded99f1-3b43-4e81-80fa-21ee12b18e23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:01.899991 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-8ded99f1-3b43-4e81-80fa-21ee12b18e23 None None] MFA Rules not processed for user `9b247fad73ea4f34af0db811be6c0048`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:01.901615 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-8ded99f1-3b43-4e81-80fa-21ee12b18e23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:01.981433 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1273/2559] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:01 2022] POST /identity/v3/auth/tokens => generated 3274 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:23:01.986374 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bde4ca0-4e0a-4bca-b641-b310c5931929 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:02.056784 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bde4ca0-4e0a-4bca-b641-b310c5931929 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:02.062375 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53aa1a6-dfc1-4eca-84c7-2caef94f48c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:02.062996 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53aa1a6-dfc1-4eca-84c7-2caef94f48c4 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:02.063277 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53aa1a6-dfc1-4eca-84c7-2caef94f48c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:02.063554 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53aa1a6-dfc1-4eca-84c7-2caef94f48c4 admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:02.064226 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53aa1a6-dfc1-4eca-84c7-2caef94f48c4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:02.066130 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53aa1a6-dfc1-4eca-84c7-2caef94f48c4 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:02.099084 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1287/2560] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:23:01 2022] POST /identity/v3/projects => generated 386 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 16:23:02.100908 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ded99f1-3b43-4e81-80fa-21ee12b18e23 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:02.114537 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ded99f1-3b43-4e81-80fa-21ee12b18e23 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:02.171305 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:02.178212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8dc3857-c220-4d30-9784-43e8a0c7d374 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:02.179187 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dc3857-c220-4d30-9784-43e8a0c7d374 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:02.179720 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dc3857-c220-4d30-9784-43e8a0c7d374 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:02.180270 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dc3857-c220-4d30-9784-43e8a0c7d374 admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:02.181103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8dc3857-c220-4d30-9784-43e8a0c7d374 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:02.183860 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8dc3857-c220-4d30-9784-43e8a0c7d374 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:02.208927 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1274/2561] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:23:02 2022] POST /identity/v3/users => generated 294 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:23:02.212927 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53aa1a6-dfc1-4eca-84c7-2caef94f48c4 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:02.224819 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53aa1a6-dfc1-4eca-84c7-2caef94f48c4 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:02.229645 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4de5eb-9ccb-40b9-b190-5818e084c498 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:02.230257 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4de5eb-9ccb-40b9-b190-5818e084c498 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:02.231309 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4de5eb-9ccb-40b9-b190-5818e084c498 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:02.231742 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4de5eb-9ccb-40b9-b190-5818e084c498 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:02.232343 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c4de5eb-9ccb-40b9-b190-5818e084c498 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:02.233942 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c4de5eb-9ccb-40b9-b190-5818e084c498 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:02.245735 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1288/2562] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:23:02 2022] GET /identity/v3/roles => generated 1427 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:23:02.250313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8dc3857-c220-4d30-9784-43e8a0c7d374 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:02.274958 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8dc3857-c220-4d30-9784-43e8a0c7d374 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:02.283609 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff8c3b4-0c55-43d5-9584-001733cbafb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:02.284502 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff8c3b4-0c55-43d5-9584-001733cbafb9 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:02.284846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff8c3b4-0c55-43d5-9584-001733cbafb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:02.285186 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff8c3b4-0c55-43d5-9584-001733cbafb9 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:02.285911 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff8c3b4-0c55-43d5-9584-001733cbafb9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:02.287760 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff8c3b4-0c55-43d5-9584-001733cbafb9 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:02.302454 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1275/2563] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:23:02 2022] GET /identity/v3/roles => generated 1427 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:23:02.306741 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4de5eb-9ccb-40b9-b190-5818e084c498 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:02.322170 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4de5eb-9ccb-40b9-b190-5818e084c498 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:02.332692 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ffbabc-7903-4cb4-9461-5e5295a98d60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:02.333471 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ffbabc-7903-4cb4-9461-5e5295a98d60 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:02.333825 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ffbabc-7903-4cb4-9461-5e5295a98d60 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:02.334169 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ffbabc-7903-4cb4-9461-5e5295a98d60 admin admin] PATH_INFO: `/v3/projects/d9ced8e57c7e443ebf30ab0f07ad9c51/users/4a7e278bef2841cc98f9674892f283cd/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:02.353471 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ffbabc-7903-4cb4-9461-5e5295a98d60 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d9ced8e57c7e443ebf30ab0f07ad9c51, user_id=4a7e278bef2841cc98f9674892f283cd, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:02.357082 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ffbabc-7903-4cb4-9461-5e5295a98d60 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:02.375132 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1289/2564] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:23:02 2022] PUT /identity/v3/projects/d9ced8e57c7e443ebf30ab0f07ad9c51/users/4a7e278bef2841cc98f9674892f283cd/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:02.385665 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e541712c-b78d-4bb6-9516-3f5459c04be2 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:02.385895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e541712c-b78d-4bb6-9516-3f5459c04be2 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:02.386104 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e541712c-b78d-4bb6-9516-3f5459c04be2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:02.429748 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-e541712c-b78d-4bb6-9516-3f5459c04be2 None None] MFA Rules not processed for user `4a7e278bef2841cc98f9674892f283cd`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:02.431141 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-e541712c-b78d-4bb6-9516-3f5459c04be2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:02.548963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1276/2565] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:02 2022] POST /identity/v3/auth/tokens => generated 3272 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:23:02.551118 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ffbabc-7903-4cb4-9461-5e5295a98d60 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:02.599301 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=91,keystone:DELETE=4,keystone:INSERT=5 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:02.722240 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ffbabc-7903-4cb4-9461-5e5295a98d60 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:02.742750 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5a4168-8002-494e-9e9b-21f852269372 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:02.743332 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5a4168-8002-494e-9e9b-21f852269372 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:02.743617 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5a4168-8002-494e-9e9b-21f852269372 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:02.743953 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5a4168-8002-494e-9e9b-21f852269372 admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:02.744620 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5a4168-8002-494e-9e9b-21f852269372 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:02.748194 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5a4168-8002-494e-9e9b-21f852269372 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:02.791244 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1290/2566] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:23:02 2022] POST /identity/v3/projects => generated 388 bytes in 240 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:23:02.796964 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e541712c-b78d-4bb6-9516-3f5459c04be2 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:02.811473 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e541712c-b78d-4bb6-9516-3f5459c04be2 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:02.898361 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31196129-6eee-44ae-a753-bf66a7260725 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:02.900222 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31196129-6eee-44ae-a753-bf66a7260725 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:02.900514 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31196129-6eee-44ae-a753-bf66a7260725 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:02.900799 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31196129-6eee-44ae-a753-bf66a7260725 admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:02.901471 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31196129-6eee-44ae-a753-bf66a7260725 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:02.906124 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31196129-6eee-44ae-a753-bf66a7260725 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:02.950520 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1277/2567] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:23:02 2022] POST /identity/v3/users => generated 301 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:23:02.951189 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5a4168-8002-494e-9e9b-21f852269372 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:02.963228 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5a4168-8002-494e-9e9b-21f852269372 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:02.971548 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a225bfa-77cd-4e1d-8a8a-913c8c9866cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:02.972247 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a225bfa-77cd-4e1d-8a8a-913c8c9866cb admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:02.972535 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a225bfa-77cd-4e1d-8a8a-913c8c9866cb admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:02.973277 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a225bfa-77cd-4e1d-8a8a-913c8c9866cb admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:02.974144 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a225bfa-77cd-4e1d-8a8a-913c8c9866cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:02.976329 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a225bfa-77cd-4e1d-8a8a-913c8c9866cb admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:02.994773 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1291/2568] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:23:02 2022] GET /identity/v3/roles => generated 1427 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:23:02.996549 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31196129-6eee-44ae-a753-bf66a7260725 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:03.008054 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31196129-6eee-44ae-a753-bf66a7260725 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:03.014319 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143520a7-0d20-4714-b559-59490c108d51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:03.015278 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143520a7-0d20-4714-b559-59490c108d51 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:03.015896 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143520a7-0d20-4714-b559-59490c108d51 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:03.016172 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143520a7-0d20-4714-b559-59490c108d51 admin admin] PATH_INFO: `/v3/projects/fbee56318f364d16b12060ef17e9c2e9/users/7dc6635afc8a436d93078c2b80ff2c81/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:03.035526 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143520a7-0d20-4714-b559-59490c108d51 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fbee56318f364d16b12060ef17e9c2e9, user_id=7dc6635afc8a436d93078c2b80ff2c81, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:03.038731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143520a7-0d20-4714-b559-59490c108d51 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:03.061332 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1278/2569] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:23:02 2022] PUT /identity/v3/projects/fbee56318f364d16b12060ef17e9c2e9/users/7dc6635afc8a436d93078c2b80ff2c81/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:03.062178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a225bfa-77cd-4e1d-8a8a-913c8c9866cb admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:03.073567 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a225bfa-77cd-4e1d-8a8a-913c8c9866cb admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:03.158058 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c92f788-e2c4-4e5f-95ae-5f4f59426289 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:03.158732 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c92f788-e2c4-4e5f-95ae-5f4f59426289 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:03.159013 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c92f788-e2c4-4e5f-95ae-5f4f59426289 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:03.159292 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c92f788-e2c4-4e5f-95ae-5f4f59426289 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:03.160000 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c92f788-e2c4-4e5f-95ae-5f4f59426289 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:03.161648 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c92f788-e2c4-4e5f-95ae-5f4f59426289 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:03.184634 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143520a7-0d20-4714-b559-59490c108d51 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:03.197545 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1292/2570] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:23:03 2022] GET /identity/v3/roles => generated 1427 bytes in 131 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 16:23:03.215343 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143520a7-0d20-4714-b559-59490c108d51 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:03.234087 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2d2786-cb12-484d-97ab-cde94707e6f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:03.234799 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2d2786-cb12-484d-97ab-cde94707e6f9 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:03.235059 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2d2786-cb12-484d-97ab-cde94707e6f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:03.235330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2d2786-cb12-484d-97ab-cde94707e6f9 admin admin] PATH_INFO: `/v3/projects/fbee56318f364d16b12060ef17e9c2e9/users/7dc6635afc8a436d93078c2b80ff2c81/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:03.239663 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce2d2786-cb12-484d-97ab-cde94707e6f9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fbee56318f364d16b12060ef17e9c2e9, user_id=7dc6635afc8a436d93078c2b80ff2c81, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:03.242797 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce2d2786-cb12-484d-97ab-cde94707e6f9 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:03.269741 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1279/2571] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:23:03 2022] PUT /identity/v3/projects/fbee56318f364d16b12060ef17e9c2e9/users/7dc6635afc8a436d93078c2b80ff2c81/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 86 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:03.285099 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127612a3-8b9a-4266-a890-f44e3317d180 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:03.285621 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127612a3-8b9a-4266-a890-f44e3317d180 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:03.285727 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127612a3-8b9a-4266-a890-f44e3317d180 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:03.320099 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-127612a3-8b9a-4266-a890-f44e3317d180 None None] MFA Rules not processed for user `7dc6635afc8a436d93078c2b80ff2c81`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:03.321902 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-127612a3-8b9a-4266-a890-f44e3317d180 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:03.412063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1293/2572] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:03 2022] POST /identity/v3/auth/tokens => generated 3341 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:23:03.419766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2d2786-cb12-484d-97ab-cde94707e6f9 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:03.421975 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-ce2d2786-cb12-484d-97ab-cde94707e6f9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:03.453031 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:03.522215 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2d2786-cb12-484d-97ab-cde94707e6f9 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:03.528668 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e59cae-f831-44f9-a3f7-7ad7bd1d55ca service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:03.529598 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e59cae-f831-44f9-a3f7-7ad7bd1d55ca service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:03.529598 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e59cae-f831-44f9-a3f7-7ad7bd1d55ca service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:03.529598 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e59cae-f831-44f9-a3f7-7ad7bd1d55ca service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:03.541378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e59cae-f831-44f9-a3f7-7ad7bd1d55ca service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:03.543779 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e59cae-f831-44f9-a3f7-7ad7bd1d55ca service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:03.558095 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1280/2573] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:23:03 2022] GET /identity/v3/auth/tokens => generated 3341 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:23:04.350829 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127612a3-8b9a-4266-a890-f44e3317d180 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:04.377355 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127612a3-8b9a-4266-a890-f44e3317d180 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:04.384750 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74f3947-1ad7-4782-93a1-53a0fe0e0e0d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:04.386298 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74f3947-1ad7-4782-93a1-53a0fe0e0e0d service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:04.387567 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74f3947-1ad7-4782-93a1-53a0fe0e0e0d service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:04.388881 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74f3947-1ad7-4782-93a1-53a0fe0e0e0d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:04.404007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:04.491532 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a74f3947-1ad7-4782-93a1-53a0fe0e0e0d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:04.493296 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a74f3947-1ad7-4782-93a1-53a0fe0e0e0d service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:04.504964 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1294/2574] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:23:04 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:23:04.941417 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a50fa7-9751-4830-98bd-34b435aefdbc None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:04.941952 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a50fa7-9751-4830-98bd-34b435aefdbc None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:04.942129 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a50fa7-9751-4830-98bd-34b435aefdbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:04.974402 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-93a50fa7-9751-4830-98bd-34b435aefdbc None None] MFA Rules not processed for user `fe64f798929444dd838825dc0d2700cf`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:04.975965 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-93a50fa7-9751-4830-98bd-34b435aefdbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:05.063554 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1281/2575] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:04 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:23:05.064751 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74f3947-1ad7-4782-93a1-53a0fe0e0e0d service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:05.066220 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-a74f3947-1ad7-4782-93a1-53a0fe0e0e0d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:05.139261 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74f3947-1ad7-4782-93a1-53a0fe0e0e0d service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:05.143939 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c9e4b0-eeba-4798-8f32-59428aeecaf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:05.144548 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c9e4b0-eeba-4798-8f32-59428aeecaf0 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:05.144825 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c9e4b0-eeba-4798-8f32-59428aeecaf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:05.145094 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c9e4b0-eeba-4798-8f32-59428aeecaf0 admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:05.145715 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c9e4b0-eeba-4798-8f32-59428aeecaf0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:05.147621 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c9e4b0-eeba-4798-8f32-59428aeecaf0 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:05.176290 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1295/2576] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:23:05 2022] POST /identity/v3/projects => generated 380 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:23:05.180573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a50fa7-9751-4830-98bd-34b435aefdbc None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:05.193001 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a50fa7-9751-4830-98bd-34b435aefdbc None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:05.255410 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67dc14c4-8619-4876-84df-cca3cf9a521e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:05.256060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67dc14c4-8619-4876-84df-cca3cf9a521e admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:05.256330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67dc14c4-8619-4876-84df-cca3cf9a521e admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:05.256666 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67dc14c4-8619-4876-84df-cca3cf9a521e admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:05.257247 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67dc14c4-8619-4876-84df-cca3cf9a521e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:05.259233 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67dc14c4-8619-4876-84df-cca3cf9a521e admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:05.283594 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1282/2577] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:23:05 2022] POST /identity/v3/users => generated 297 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:23:05.287936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c9e4b0-eeba-4798-8f32-59428aeecaf0 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:05.298449 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c9e4b0-eeba-4798-8f32-59428aeecaf0 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:05.303785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d14cb1f-e0c9-4e24-9567-548da0dfa023 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:05.304417 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d14cb1f-e0c9-4e24-9567-548da0dfa023 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:05.304734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d14cb1f-e0c9-4e24-9567-548da0dfa023 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:05.305150 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d14cb1f-e0c9-4e24-9567-548da0dfa023 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:05.305899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d14cb1f-e0c9-4e24-9567-548da0dfa023 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:05.308050 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d14cb1f-e0c9-4e24-9567-548da0dfa023 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:05.319920 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1296/2578] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:23:05 2022] GET /identity/v3/roles => generated 1427 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 16:23:05.324897 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67dc14c4-8619-4876-84df-cca3cf9a521e admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:05.334709 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67dc14c4-8619-4876-84df-cca3cf9a521e admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:05.340004 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd092aa-04d7-4263-bed6-c58fa287ba52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:05.340871 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd092aa-04d7-4263-bed6-c58fa287ba52 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:05.341150 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd092aa-04d7-4263-bed6-c58fa287ba52 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:05.341422 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd092aa-04d7-4263-bed6-c58fa287ba52 admin admin] PATH_INFO: `/v3/projects/c9935dd42b694ec68fef78a45b54709c/users/1956cb5416904bf4b30243e1318249d9/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:05.356388 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd092aa-04d7-4263-bed6-c58fa287ba52 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c9935dd42b694ec68fef78a45b54709c, user_id=1956cb5416904bf4b30243e1318249d9, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:05.359137 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd092aa-04d7-4263-bed6-c58fa287ba52 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:05.373463 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1283/2579] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:23:05 2022] PUT /identity/v3/projects/c9935dd42b694ec68fef78a45b54709c/users/1956cb5416904bf4b30243e1318249d9/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:05.377370 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d14cb1f-e0c9-4e24-9567-548da0dfa023 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:05.389128 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d14cb1f-e0c9-4e24-9567-548da0dfa023 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:05.453000 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2289b7-4cb1-4efb-874c-e51d81d7d8be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:05.453609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2289b7-4cb1-4efb-874c-e51d81d7d8be admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:05.453888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2289b7-4cb1-4efb-874c-e51d81d7d8be admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:05.454159 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2289b7-4cb1-4efb-874c-e51d81d7d8be admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:05.454796 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2289b7-4cb1-4efb-874c-e51d81d7d8be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:05.456413 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2289b7-4cb1-4efb-874c-e51d81d7d8be admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:05.467452 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1297/2580] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:23:05 2022] GET /identity/v3/roles => generated 1427 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:23:05.471847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd092aa-04d7-4263-bed6-c58fa287ba52 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:05.482840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd092aa-04d7-4263-bed6-c58fa287ba52 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:05.487435 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56de22d-0474-4f50-aac0-54ca9931dae8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:05.488101 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56de22d-0474-4f50-aac0-54ca9931dae8 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:05.488380 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56de22d-0474-4f50-aac0-54ca9931dae8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:05.488652 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56de22d-0474-4f50-aac0-54ca9931dae8 admin admin] PATH_INFO: `/v3/projects/c9935dd42b694ec68fef78a45b54709c/users/1956cb5416904bf4b30243e1318249d9/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:05.490657 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56de22d-0474-4f50-aac0-54ca9931dae8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c9935dd42b694ec68fef78a45b54709c, user_id=1956cb5416904bf4b30243e1318249d9, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:05.493355 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56de22d-0474-4f50-aac0-54ca9931dae8 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:05.505038 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1284/2581] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:23:05 2022] PUT /identity/v3/projects/c9935dd42b694ec68fef78a45b54709c/users/1956cb5416904bf4b30243e1318249d9/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:09.106368 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2289b7-4cb1-4efb-874c-e51d81d7d8be admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:09.161595 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:09.276903 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2289b7-4cb1-4efb-874c-e51d81d7d8be admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:09.386031 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70a1a70-9726-4dd1-88ef-d9edb9d9b870 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:09.387176 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70a1a70-9726-4dd1-88ef-d9edb9d9b870 service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:09.387473 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70a1a70-9726-4dd1-88ef-d9edb9d9b870 service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:09.387780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70a1a70-9726-4dd1-88ef-d9edb9d9b870 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:09.521612 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70a1a70-9726-4dd1-88ef-d9edb9d9b870 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:09.523095 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70a1a70-9726-4dd1-88ef-d9edb9d9b870 service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:09.544973 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1298/2582] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:23:09 2022] GET /identity/v3/auth/tokens => generated 3272 bytes in 439 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:23:11.171799 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d711e1f-2740-446e-9c49-bc8c625039b9 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:11.173156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d711e1f-2740-446e-9c49-bc8c625039b9 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:11.173755 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d711e1f-2740-446e-9c49-bc8c625039b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:11.192321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:11.220722 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-3d711e1f-2740-446e-9c49-bc8c625039b9 None None] MFA Rules not processed for user `1956cb5416904bf4b30243e1318249d9`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:11.222093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-3d711e1f-2740-446e-9c49-bc8c625039b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:11.323674 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1285/2583] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:11 2022] POST /identity/v3/auth/tokens => generated 3333 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:23:11.348447 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70a1a70-9726-4dd1-88ef-d9edb9d9b870 service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:11.349911 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-f70a1a70-9726-4dd1-88ef-d9edb9d9b870 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:11.446765 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70a1a70-9726-4dd1-88ef-d9edb9d9b870 service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:11.454459 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111815d6-4063-48c5-9cf0-36c476d3a29a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:11.455093 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111815d6-4063-48c5-9cf0-36c476d3a29a service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:11.455436 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111815d6-4063-48c5-9cf0-36c476d3a29a service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:11.455743 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111815d6-4063-48c5-9cf0-36c476d3a29a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:11.488078 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:11.524512 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111815d6-4063-48c5-9cf0-36c476d3a29a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:11.526125 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111815d6-4063-48c5-9cf0-36c476d3a29a service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:11.536431 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1299/2584] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:23:11 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 188 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:23:12.481697 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d711e1f-2740-446e-9c49-bc8c625039b9 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:12.579185 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d711e1f-2740-446e-9c49-bc8c625039b9 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:12.585084 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8c6463-8458-416d-ad09-ceceae56af3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:12.586065 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8c6463-8458-416d-ad09-ceceae56af3e admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:12.586344 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8c6463-8458-416d-ad09-ceceae56af3e admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:12.586642 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8c6463-8458-416d-ad09-ceceae56af3e admin admin] PATH_INFO: `/v3/users/9b247fad73ea4f34af0db811be6c0048` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:12.588360 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8c6463-8458-416d-ad09-ceceae56af3e admin admin] RBAC: Authorizing `identity:delete_user(user_id=9b247fad73ea4f34af0db811be6c0048)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:12.591452 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8c6463-8458-416d-ad09-ceceae56af3e admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:12.675681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-7e8c6463-8458-416d-ad09-ceceae56af3e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9b247fad73ea4f34af0db811be6c0048'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:12.675681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-7e8c6463-8458-416d-ad09-ceceae56af3e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9b247fad73ea4f34af0db811be6c0048'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:12.690462 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-7e8c6463-8458-416d-ad09-ceceae56af3e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9b247fad73ea4f34af0db811be6c0048'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:12.724170 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-7e8c6463-8458-416d-ad09-ceceae56af3e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9b247fad73ea4f34af0db811be6c0048'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:12.726482 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-7e8c6463-8458-416d-ad09-ceceae56af3e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9b247fad73ea4f34af0db811be6c0048'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:12.750812 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1286/2585] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:23:12 2022] DELETE /identity/v3/users/9b247fad73ea4f34af0db811be6c0048 => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:12.755779 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0790a57a-1ec8-40da-9e7b-c1637cd20de7 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:12.758813 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0790a57a-1ec8-40da-9e7b-c1637cd20de7 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:12.759813 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0790a57a-1ec8-40da-9e7b-c1637cd20de7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:12.806780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-0790a57a-1ec8-40da-9e7b-c1637cd20de7 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:12.808400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-0790a57a-1ec8-40da-9e7b-c1637cd20de7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:12.892218 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1300/2586] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:12 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:23:12.904047 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8c6463-8458-416d-ad09-ceceae56af3e admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:12.907237 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-7e8c6463-8458-416d-ad09-ceceae56af3e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:12.920084 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:12.983090 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8c6463-8458-416d-ad09-ceceae56af3e admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:12.988036 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc3c4f6-5ad6-4498-b545-4c6f4dee4866 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:12.988650 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc3c4f6-5ad6-4498-b545-4c6f4dee4866 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:12.988930 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc3c4f6-5ad6-4498-b545-4c6f4dee4866 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:12.989198 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc3c4f6-5ad6-4498-b545-4c6f4dee4866 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:12.999578 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc3c4f6-5ad6-4498-b545-4c6f4dee4866 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:13.000840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc3c4f6-5ad6-4498-b545-4c6f4dee4866 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:13.016335 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1287/2587] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:23:12 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 16:23:13.391336 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0790a57a-1ec8-40da-9e7b-c1637cd20de7 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:13.393072 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-0790a57a-1ec8-40da-9e7b-c1637cd20de7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:13.408063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0790a57a-1ec8-40da-9e7b-c1637cd20de7 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:13.413279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb43614f-8161-45bf-b413-2d5a49e71cb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:13.414261 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb43614f-8161-45bf-b413-2d5a49e71cb6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:13.414550 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb43614f-8161-45bf-b413-2d5a49e71cb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:13.414856 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb43614f-8161-45bf-b413-2d5a49e71cb6 admin admin] PATH_INFO: `/v3/projects/6bd76ba8381947eebda2630779ef3b8a` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:13.416114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb43614f-8161-45bf-b413-2d5a49e71cb6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6bd76ba8381947eebda2630779ef3b8a)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:13.418615 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb43614f-8161-45bf-b413-2d5a49e71cb6 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:13.473455 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-fb43614f-8161-45bf-b413-2d5a49e71cb6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6bd76ba8381947eebda2630779ef3b8a', 'request_id': 'req-fb43614f-8161-45bf-b413-2d5a49e71cb6'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:13.482054 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-fb43614f-8161-45bf-b413-2d5a49e71cb6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6bd76ba8381947eebda2630779ef3b8a', 'request_id': 'req-fb43614f-8161-45bf-b413-2d5a49e71cb6'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:13.487931 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-fb43614f-8161-45bf-b413-2d5a49e71cb6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6bd76ba8381947eebda2630779ef3b8a', 'request_id': 'req-fb43614f-8161-45bf-b413-2d5a49e71cb6'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:13.504677 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-fb43614f-8161-45bf-b413-2d5a49e71cb6 admin admin] The token cache is being invalidate because project 6bd76ba8381947eebda2630779ef3b8a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:23:13.505019 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-fb43614f-8161-45bf-b413-2d5a49e71cb6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:13.505639 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-fb43614f-8161-45bf-b413-2d5a49e71cb6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:13.507180 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1301/2588] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:23:13 2022] DELETE /identity/v3/projects/6bd76ba8381947eebda2630779ef3b8a => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:13.511323 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc3c4f6-5ad6-4498-b545-4c6f4dee4866 service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:13.522346 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc3c4f6-5ad6-4498-b545-4c6f4dee4866 service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:13.590854 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d896f532-b0df-4a91-ac90-7ae98a9b35fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:13.591507 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d896f532-b0df-4a91-ac90-7ae98a9b35fb admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:13.591837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d896f532-b0df-4a91-ac90-7ae98a9b35fb admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:13.592119 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d896f532-b0df-4a91-ac90-7ae98a9b35fb admin admin] PATH_INFO: `/v3/users/4a7e278bef2841cc98f9674892f283cd` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:13.593429 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d896f532-b0df-4a91-ac90-7ae98a9b35fb admin admin] RBAC: Authorizing `identity:delete_user(user_id=4a7e278bef2841cc98f9674892f283cd)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:13.595416 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d896f532-b0df-4a91-ac90-7ae98a9b35fb admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:13.661835 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d896f532-b0df-4a91-ac90-7ae98a9b35fb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4a7e278bef2841cc98f9674892f283cd'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:13.662624 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d896f532-b0df-4a91-ac90-7ae98a9b35fb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4a7e278bef2841cc98f9674892f283cd'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:13.671128 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d896f532-b0df-4a91-ac90-7ae98a9b35fb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4a7e278bef2841cc98f9674892f283cd'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:13.687608 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d896f532-b0df-4a91-ac90-7ae98a9b35fb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4a7e278bef2841cc98f9674892f283cd'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:13.688438 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d896f532-b0df-4a91-ac90-7ae98a9b35fb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4a7e278bef2841cc98f9674892f283cd'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:13.706901 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1288/2589] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:23:13 2022] DELETE /identity/v3/users/4a7e278bef2841cc98f9674892f283cd => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:14.088187 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb43614f-8161-45bf-b413-2d5a49e71cb6 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:14.091651 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-fb43614f-8161-45bf-b413-2d5a49e71cb6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:14.108081 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb43614f-8161-45bf-b413-2d5a49e71cb6 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:14.196437 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:14.212358 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db2ac5b4-5572-4400-8bf1-e2d8695a1113 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:14.213600 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db2ac5b4-5572-4400-8bf1-e2d8695a1113 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:14.213839 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db2ac5b4-5572-4400-8bf1-e2d8695a1113 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:14.214589 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db2ac5b4-5572-4400-8bf1-e2d8695a1113 admin admin] PATH_INFO: `/v3/projects/d9ced8e57c7e443ebf30ab0f07ad9c51` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:14.216602 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db2ac5b4-5572-4400-8bf1-e2d8695a1113 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d9ced8e57c7e443ebf30ab0f07ad9c51)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:14.220447 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db2ac5b4-5572-4400-8bf1-e2d8695a1113 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:14.306443 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-db2ac5b4-5572-4400-8bf1-e2d8695a1113 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd9ced8e57c7e443ebf30ab0f07ad9c51', 'request_id': 'req-db2ac5b4-5572-4400-8bf1-e2d8695a1113'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:14.341058 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-db2ac5b4-5572-4400-8bf1-e2d8695a1113 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd9ced8e57c7e443ebf30ab0f07ad9c51', 'request_id': 'req-db2ac5b4-5572-4400-8bf1-e2d8695a1113'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:14.362878 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-db2ac5b4-5572-4400-8bf1-e2d8695a1113 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd9ced8e57c7e443ebf30ab0f07ad9c51', 'request_id': 'req-db2ac5b4-5572-4400-8bf1-e2d8695a1113'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:14.398014 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-db2ac5b4-5572-4400-8bf1-e2d8695a1113 admin admin] The token cache is being invalidate because project d9ced8e57c7e443ebf30ab0f07ad9c51 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:23:14.398735 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-db2ac5b4-5572-4400-8bf1-e2d8695a1113 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:14.400560 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-db2ac5b4-5572-4400-8bf1-e2d8695a1113 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:14.408007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1302/2590] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:23:14 2022] DELETE /identity/v3/projects/d9ced8e57c7e443ebf30ab0f07ad9c51 => generated 0 bytes in 320 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:14.412227 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d896f532-b0df-4a91-ac90-7ae98a9b35fb admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:14.413770 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-d896f532-b0df-4a91-ac90-7ae98a9b35fb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:14.430326 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d896f532-b0df-4a91-ac90-7ae98a9b35fb admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:14.480094 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:14.544877 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807b3432-3442-4822-9bc3-0a3ef867df02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:14.545608 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807b3432-3442-4822-9bc3-0a3ef867df02 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:14.545990 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807b3432-3442-4822-9bc3-0a3ef867df02 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:14.546324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807b3432-3442-4822-9bc3-0a3ef867df02 admin admin] PATH_INFO: `/v3/users/7dc6635afc8a436d93078c2b80ff2c81` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:14.547522 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807b3432-3442-4822-9bc3-0a3ef867df02 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7dc6635afc8a436d93078c2b80ff2c81)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:14.550852 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807b3432-3442-4822-9bc3-0a3ef867df02 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:14.615537 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-807b3432-3442-4822-9bc3-0a3ef867df02 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7dc6635afc8a436d93078c2b80ff2c81'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:14.616733 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-807b3432-3442-4822-9bc3-0a3ef867df02 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7dc6635afc8a436d93078c2b80ff2c81'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:14.627830 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-807b3432-3442-4822-9bc3-0a3ef867df02 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7dc6635afc8a436d93078c2b80ff2c81'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:14.650189 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-807b3432-3442-4822-9bc3-0a3ef867df02 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7dc6635afc8a436d93078c2b80ff2c81'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:14.654397 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-807b3432-3442-4822-9bc3-0a3ef867df02 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7dc6635afc8a436d93078c2b80ff2c81'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:14.673385 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1289/2591] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:23:14 2022] DELETE /identity/v3/users/7dc6635afc8a436d93078c2b80ff2c81 => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:14.987548 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db2ac5b4-5572-4400-8bf1-e2d8695a1113 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:14.989047 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-db2ac5b4-5572-4400-8bf1-e2d8695a1113 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:15.006956 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db2ac5b4-5572-4400-8bf1-e2d8695a1113 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:15.069245 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e399b9-cb75-4589-9014-50041c6eb867 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:15.069887 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e399b9-cb75-4589-9014-50041c6eb867 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:15.070172 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e399b9-cb75-4589-9014-50041c6eb867 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:15.070450 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e399b9-cb75-4589-9014-50041c6eb867 admin admin] PATH_INFO: `/v3/projects/fbee56318f364d16b12060ef17e9c2e9` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:15.071600 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e399b9-cb75-4589-9014-50041c6eb867 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fbee56318f364d16b12060ef17e9c2e9)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:15.073543 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e399b9-cb75-4589-9014-50041c6eb867 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:15.123255 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-e0e399b9-cb75-4589-9014-50041c6eb867 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fbee56318f364d16b12060ef17e9c2e9', 'request_id': 'req-e0e399b9-cb75-4589-9014-50041c6eb867'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:15.129478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-e0e399b9-cb75-4589-9014-50041c6eb867 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fbee56318f364d16b12060ef17e9c2e9', 'request_id': 'req-e0e399b9-cb75-4589-9014-50041c6eb867'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:15.135226 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-e0e399b9-cb75-4589-9014-50041c6eb867 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fbee56318f364d16b12060ef17e9c2e9', 'request_id': 'req-e0e399b9-cb75-4589-9014-50041c6eb867'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:15.148379 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-e0e399b9-cb75-4589-9014-50041c6eb867 admin admin] The token cache is being invalidate because project fbee56318f364d16b12060ef17e9c2e9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:23:15.148542 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-e0e399b9-cb75-4589-9014-50041c6eb867 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:15.149184 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-e0e399b9-cb75-4589-9014-50041c6eb867 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:15.150665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1303/2592] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:23:14 2022] DELETE /identity/v3/projects/fbee56318f364d16b12060ef17e9c2e9 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:15.156963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f159442a-4791-4cab-9fca-568811aa912c None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:15.157250 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f159442a-4791-4cab-9fca-568811aa912c None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:15.157531 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f159442a-4791-4cab-9fca-568811aa912c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:15.176916 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-f159442a-4791-4cab-9fca-568811aa912c None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:15.178028 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-f159442a-4791-4cab-9fca-568811aa912c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:15.240389 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1290/2593] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:15 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 16:23:15.249332 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129b98c0-afe9-41cb-b018-e9f953d20b8a None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:15.249623 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129b98c0-afe9-41cb-b018-e9f953d20b8a None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:15.249901 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129b98c0-afe9-41cb-b018-e9f953d20b8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:15.269601 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-129b98c0-afe9-41cb-b018-e9f953d20b8a None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:15.270806 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-129b98c0-afe9-41cb-b018-e9f953d20b8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:15.281934 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1304/2594] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:15 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:23:15.286185 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f159442a-4791-4cab-9fca-568811aa912c None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:15.297383 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f159442a-4791-4cab-9fca-568811aa912c None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:15.303066 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b5998b-afbe-497f-8e27-33d8034c4b1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:15.303679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b5998b-afbe-497f-8e27-33d8034c4b1e admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:15.303964 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b5998b-afbe-497f-8e27-33d8034c4b1e admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:15.304241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b5998b-afbe-497f-8e27-33d8034c4b1e admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:15.304948 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b5998b-afbe-497f-8e27-33d8034c4b1e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 16:23:15.305258 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b5998b-afbe-497f-8e27-33d8034c4b1e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:15.307086 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b5998b-afbe-497f-8e27-33d8034c4b1e admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:15.320847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1291/2595] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 16:23:15 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 16:23:15.325295 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129b98c0-afe9-41cb-b018-e9f953d20b8a None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:15.334668 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129b98c0-afe9-41cb-b018-e9f953d20b8a None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:15.339578 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-330d7177-bf00-4d47-84ae-2cedc760fda3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:15.340236 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330d7177-bf00-4d47-84ae-2cedc760fda3 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:15.340519 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330d7177-bf00-4d47-84ae-2cedc760fda3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:15.340793 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330d7177-bf00-4d47-84ae-2cedc760fda3 admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:15.341416 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-330d7177-bf00-4d47-84ae-2cedc760fda3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:15.343315 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-330d7177-bf00-4d47-84ae-2cedc760fda3 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:15.348225 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:15.374372 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1305/2596] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:23:15 2022] POST /identity/v3/projects => generated 372 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:23:15.378501 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b5998b-afbe-497f-8e27-33d8034c4b1e admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:15.388586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b5998b-afbe-497f-8e27-33d8034c4b1e admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:15.426324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:15.467111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc7247f-243c-4ad8-ac3d-ea4eb302273b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:15.467951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc7247f-243c-4ad8-ac3d-ea4eb302273b admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:15.472370 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc7247f-243c-4ad8-ac3d-ea4eb302273b admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:15.472370 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc7247f-243c-4ad8-ac3d-ea4eb302273b admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:15.472370 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc7247f-243c-4ad8-ac3d-ea4eb302273b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:15.472370 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc7247f-243c-4ad8-ac3d-ea4eb302273b admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:15.500506 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1292/2597] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:23:15 2022] POST /identity/v3/users => generated 287 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 16:23:15.504634 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-330d7177-bf00-4d47-84ae-2cedc760fda3 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:15.516400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-330d7177-bf00-4d47-84ae-2cedc760fda3 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:15.521885 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70a8b86-9b9c-4f70-9255-b93561acc196 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:15.522560 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70a8b86-9b9c-4f70-9255-b93561acc196 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:15.522859 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70a8b86-9b9c-4f70-9255-b93561acc196 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:15.523118 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70a8b86-9b9c-4f70-9255-b93561acc196 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:15.523813 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70a8b86-9b9c-4f70-9255-b93561acc196 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:15.525454 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70a8b86-9b9c-4f70-9255-b93561acc196 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:15.537763 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1306/2598] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:23:15 2022] GET /identity/v3/roles => generated 1427 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:23:15.542651 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc7247f-243c-4ad8-ac3d-ea4eb302273b admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:15.558664 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc7247f-243c-4ad8-ac3d-ea4eb302273b admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:15.563952 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53458d4-dfb1-49d1-af20-6823b8506d6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:15.564662 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53458d4-dfb1-49d1-af20-6823b8506d6b admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:15.565030 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53458d4-dfb1-49d1-af20-6823b8506d6b admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:15.565363 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53458d4-dfb1-49d1-af20-6823b8506d6b admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:15.566111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53458d4-dfb1-49d1-af20-6823b8506d6b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:15.567942 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53458d4-dfb1-49d1-af20-6823b8506d6b admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:15.583106 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1293/2599] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:23:15 2022] GET /identity/v3/roles => generated 1427 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:23:15.584199 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70a8b86-9b9c-4f70-9255-b93561acc196 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:15.597745 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70a8b86-9b9c-4f70-9255-b93561acc196 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:15.603143 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0dbb0d-4c51-4513-ac2e-cce59ac2f375 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:15.603992 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0dbb0d-4c51-4513-ac2e-cce59ac2f375 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:15.605161 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0dbb0d-4c51-4513-ac2e-cce59ac2f375 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:15.605854 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0dbb0d-4c51-4513-ac2e-cce59ac2f375 admin admin] PATH_INFO: `/v3/projects/7ac42150994247a2a2f303085ac18420/users/1be5506d52574f8184f877d6aa9e8358/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:15.633986 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b0dbb0d-4c51-4513-ac2e-cce59ac2f375 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7ac42150994247a2a2f303085ac18420, user_id=1be5506d52574f8184f877d6aa9e8358, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:15.638239 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b0dbb0d-4c51-4513-ac2e-cce59ac2f375 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:15.660793 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1307/2600] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:23:15 2022] PUT /identity/v3/projects/7ac42150994247a2a2f303085ac18420/users/1be5506d52574f8184f877d6aa9e8358/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:15.666358 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5f3e43-7360-4d69-bab6-51c9abe0542f None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:15.666648 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5f3e43-7360-4d69-bab6-51c9abe0542f None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:15.667061 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5f3e43-7360-4d69-bab6-51c9abe0542f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:15.702287 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-4f5f3e43-7360-4d69-bab6-51c9abe0542f None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:15.703795 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-4f5f3e43-7360-4d69-bab6-51c9abe0542f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:15.802336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1294/2601] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:15 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:23:15.812843 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0dbb0d-4c51-4513-ac2e-cce59ac2f375 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:15.814677 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-5b0dbb0d-4c51-4513-ac2e-cce59ac2f375 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:15.908888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0dbb0d-4c51-4513-ac2e-cce59ac2f375 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:15.913169 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edd3488-b17b-4239-9e8e-85f8f3d9ba61 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:15.913859 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edd3488-b17b-4239-9e8e-85f8f3d9ba61 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:15.914138 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edd3488-b17b-4239-9e8e-85f8f3d9ba61 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:15.914458 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edd3488-b17b-4239-9e8e-85f8f3d9ba61 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:15.926500 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5edd3488-b17b-4239-9e8e-85f8f3d9ba61 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:15.928367 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5edd3488-b17b-4239-9e8e-85f8f3d9ba61 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:15.951879 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1308/2602] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:23:15 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:23:16.671649 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7410d6-f8e8-4e4f-a17f-a6872e157322 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:16.672169 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7410d6-f8e8-4e4f-a17f-a6872e157322 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:16.672246 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7410d6-f8e8-4e4f-a17f-a6872e157322 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:16.705908 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-7e7410d6-f8e8-4e4f-a17f-a6872e157322 None None] MFA Rules not processed for user `1be5506d52574f8184f877d6aa9e8358`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:16.707017 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-7e7410d6-f8e8-4e4f-a17f-a6872e157322 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:16.815305 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1295/2603] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:16 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:23:16.819999 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edd3488-b17b-4239-9e8e-85f8f3d9ba61 service neutron] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:16.830565 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edd3488-b17b-4239-9e8e-85f8f3d9ba61 service neutron] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:16.850288 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e99e43-df55-4430-8849-e1d4878714c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:16.855012 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e99e43-df55-4430-8849-e1d4878714c2 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:16.855012 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e99e43-df55-4430-8849-e1d4878714c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:16.855012 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e99e43-df55-4430-8849-e1d4878714c2 admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:16.855012 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76e99e43-df55-4430-8849-e1d4878714c2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:16.855012 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76e99e43-df55-4430-8849-e1d4878714c2 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:16.889870 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1309/2604] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:23:16 2022] POST /identity/v3/projects => generated 374 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:23:16.894092 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7410d6-f8e8-4e4f-a17f-a6872e157322 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:16.907055 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7410d6-f8e8-4e4f-a17f-a6872e157322 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:16.953242 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:16.969664 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb62013-2306-46ca-9b58-232c2e4ae519 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:16.970259 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb62013-2306-46ca-9b58-232c2e4ae519 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:16.970538 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb62013-2306-46ca-9b58-232c2e4ae519 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:16.970806 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb62013-2306-46ca-9b58-232c2e4ae519 admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:16.971434 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb62013-2306-46ca-9b58-232c2e4ae519 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:16.973418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb62013-2306-46ca-9b58-232c2e4ae519 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:16.995405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1296/2605] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:23:16 2022] POST /identity/v3/users => generated 294 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:23:17.000035 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e99e43-df55-4430-8849-e1d4878714c2 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:17.010484 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e99e43-df55-4430-8849-e1d4878714c2 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:17.018144 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22353f1-aba9-45bf-9445-7ea1ae674b7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:17.018894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22353f1-aba9-45bf-9445-7ea1ae674b7c admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:17.019175 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22353f1-aba9-45bf-9445-7ea1ae674b7c admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:17.019442 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22353f1-aba9-45bf-9445-7ea1ae674b7c admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:17.020204 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22353f1-aba9-45bf-9445-7ea1ae674b7c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:17.021869 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22353f1-aba9-45bf-9445-7ea1ae674b7c admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:17.034793 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1310/2606] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:23:16 2022] GET /identity/v3/roles => generated 1427 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:23:17.038704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb62013-2306-46ca-9b58-232c2e4ae519 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:17.049454 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb62013-2306-46ca-9b58-232c2e4ae519 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:17.054921 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7aa9b69-1c50-4e2a-99db-195d35ea1251 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:17.055595 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7aa9b69-1c50-4e2a-99db-195d35ea1251 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:17.055927 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7aa9b69-1c50-4e2a-99db-195d35ea1251 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:17.056201 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7aa9b69-1c50-4e2a-99db-195d35ea1251 admin admin] PATH_INFO: `/v3/projects/cab13983d391495e8c069aa0b12b549d/users/d413ebe892a94ec2b473eb71735c9941/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:17.072400 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7aa9b69-1c50-4e2a-99db-195d35ea1251 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cab13983d391495e8c069aa0b12b549d, user_id=d413ebe892a94ec2b473eb71735c9941, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:17.075598 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7aa9b69-1c50-4e2a-99db-195d35ea1251 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:17.090134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1297/2607] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:23:17 2022] PUT /identity/v3/projects/cab13983d391495e8c069aa0b12b549d/users/d413ebe892a94ec2b473eb71735c9941/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:17.094322 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22353f1-aba9-45bf-9445-7ea1ae674b7c admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:17.104574 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22353f1-aba9-45bf-9445-7ea1ae674b7c admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:17.182154 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3a39d2-69a2-4a2b-8c8e-81bbadf853e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:17.182849 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3a39d2-69a2-4a2b-8c8e-81bbadf853e5 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:17.183142 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3a39d2-69a2-4a2b-8c8e-81bbadf853e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:17.184086 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3a39d2-69a2-4a2b-8c8e-81bbadf853e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:17.185109 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3a39d2-69a2-4a2b-8c8e-81bbadf853e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:17.186701 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3a39d2-69a2-4a2b-8c8e-81bbadf853e5 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:17.197923 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1311/2608] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:23:17 2022] GET /identity/v3/roles => generated 1427 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:23:17.202821 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7aa9b69-1c50-4e2a-99db-195d35ea1251 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:17.213465 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7aa9b69-1c50-4e2a-99db-195d35ea1251 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:17.219448 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b511d9e0-eecd-43ef-992c-817d94392ccf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:17.220136 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b511d9e0-eecd-43ef-992c-817d94392ccf admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:17.220413 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b511d9e0-eecd-43ef-992c-817d94392ccf admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:17.220683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b511d9e0-eecd-43ef-992c-817d94392ccf admin admin] PATH_INFO: `/v3/projects/cab13983d391495e8c069aa0b12b549d/users/d413ebe892a94ec2b473eb71735c9941/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:17.222690 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b511d9e0-eecd-43ef-992c-817d94392ccf admin admin] RBAC: Authorizing `identity:create_grant(project_id=cab13983d391495e8c069aa0b12b549d, user_id=d413ebe892a94ec2b473eb71735c9941, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:17.225443 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b511d9e0-eecd-43ef-992c-817d94392ccf admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:17.236681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1298/2609] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:23:17 2022] PUT /identity/v3/projects/cab13983d391495e8c069aa0b12b549d/users/d413ebe892a94ec2b473eb71735c9941/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:17.906938 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7723324d-12e5-445f-b115-47a7a9c192f6 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:17.907737 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7723324d-12e5-445f-b115-47a7a9c192f6 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:17.908145 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7723324d-12e5-445f-b115-47a7a9c192f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:17.938065 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-7723324d-12e5-445f-b115-47a7a9c192f6 None None] MFA Rules not processed for user `d413ebe892a94ec2b473eb71735c9941`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:17.939376 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-7723324d-12e5-445f-b115-47a7a9c192f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:17.943901 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:18.017151 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1312/2610] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:17 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:23:18.029888 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b511d9e0-eecd-43ef-992c-817d94392ccf admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:18.032194 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-b511d9e0-eecd-43ef-992c-817d94392ccf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:18.097150 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b511d9e0-eecd-43ef-992c-817d94392ccf admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:18.100926 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8222249-914b-4779-8dd9-d5e83626179c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:18.101809 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8222249-914b-4779-8dd9-d5e83626179c service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:18.102095 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8222249-914b-4779-8dd9-d5e83626179c service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:18.102368 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8222249-914b-4779-8dd9-d5e83626179c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:18.162487 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8222249-914b-4779-8dd9-d5e83626179c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:18.163947 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8222249-914b-4779-8dd9-d5e83626179c service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:18.174259 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1299/2611] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:23:18 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:23:19.588514 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7723324d-12e5-445f-b115-47a7a9c192f6 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:19.599656 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7723324d-12e5-445f-b115-47a7a9c192f6 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:19.603359 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca39bb3d-d352-4608-9129-c5d190ecfc1b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:19.604003 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca39bb3d-d352-4608-9129-c5d190ecfc1b service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:19.604285 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca39bb3d-d352-4608-9129-c5d190ecfc1b service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:19.604559 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca39bb3d-d352-4608-9129-c5d190ecfc1b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:19.615026 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca39bb3d-d352-4608-9129-c5d190ecfc1b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:19.616534 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca39bb3d-d352-4608-9129-c5d190ecfc1b service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:19.629015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1313/2612] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:23:19 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:23:28.169742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=72,keystone:INSERT=5 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:29.624022 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=33 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:36.672858 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828497a7-4fa5-4061-93d8-6043d291e0c6 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:36.673399 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828497a7-4fa5-4061-93d8-6043d291e0c6 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:36.673399 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828497a7-4fa5-4061-93d8-6043d291e0c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:36.692964 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-828497a7-4fa5-4061-93d8-6043d291e0c6 None None] MFA Rules not processed for user `fe64f798929444dd838825dc0d2700cf`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:36.694177 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-828497a7-4fa5-4061-93d8-6043d291e0c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:36.750498 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1300/2613] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:36 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:23:37.086195 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca39bb3d-d352-4608-9129-c5d190ecfc1b service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:37.087692 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-ca39bb3d-d352-4608-9129-c5d190ecfc1b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:37.148764 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca39bb3d-d352-4608-9129-c5d190ecfc1b service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:37.153316 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd09d9c5-6393-4552-bc7c-cd7370284f00 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:37.154052 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd09d9c5-6393-4552-bc7c-cd7370284f00 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:37.154341 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd09d9c5-6393-4552-bc7c-cd7370284f00 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:37.154619 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd09d9c5-6393-4552-bc7c-cd7370284f00 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:37.156417 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-cd09d9c5-6393-4552-bc7c-cd7370284f00 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:37.167211 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd09d9c5-6393-4552-bc7c-cd7370284f00 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:37.168883 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd09d9c5-6393-4552-bc7c-cd7370284f00 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:37.177737 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1314/2614] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:23:37 2022] GET /identity/v3/auth/tokens => generated 3131 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:23:37.555807 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828497a7-4fa5-4061-93d8-6043d291e0c6 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:37.557216 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-828497a7-4fa5-4061-93d8-6043d291e0c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:37.651082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828497a7-4fa5-4061-93d8-6043d291e0c6 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:37.656934 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85da121-2118-4105-9356-acc4450427ec service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0ee0002a76c3471f819b37fef510ef99', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:37.657545 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85da121-2118-4105-9356-acc4450427ec service placement] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:37.657829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85da121-2118-4105-9356-acc4450427ec service placement] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:37.658103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85da121-2118-4105-9356-acc4450427ec service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:37.659907 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-a85da121-2118-4105-9356-acc4450427ec service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:37.673214 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85da121-2118-4105-9356-acc4450427ec service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:37.674938 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85da121-2118-4105-9356-acc4450427ec service placement] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:37.685923 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1301/2615] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:23:37 2022] GET /identity/v3/auth/tokens => generated 3318 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:23:38.161154 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd09d9c5-6393-4552-bc7c-cd7370284f00 service neutron] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:38.233078 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd09d9c5-6393-4552-bc7c-cd7370284f00 service neutron] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:38.239573 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33977334-2edb-422c-bf72-9e01ad752627 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:38.240706 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33977334-2edb-422c-bf72-9e01ad752627 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:38.241194 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33977334-2edb-422c-bf72-9e01ad752627 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:38.241473 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33977334-2edb-422c-bf72-9e01ad752627 admin admin] PATH_INFO: `/v3/users/1be5506d52574f8184f877d6aa9e8358` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:38.242641 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33977334-2edb-422c-bf72-9e01ad752627 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1be5506d52574f8184f877d6aa9e8358)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:38.244611 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33977334-2edb-422c-bf72-9e01ad752627 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:38.297818 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-33977334-2edb-422c-bf72-9e01ad752627 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1be5506d52574f8184f877d6aa9e8358'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:38.298462 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-33977334-2edb-422c-bf72-9e01ad752627 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1be5506d52574f8184f877d6aa9e8358'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:38.299056 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-33977334-2edb-422c-bf72-9e01ad752627 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1be5506d52574f8184f877d6aa9e8358'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:38.305919 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-33977334-2edb-422c-bf72-9e01ad752627 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1be5506d52574f8184f877d6aa9e8358'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:38.319446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-33977334-2edb-422c-bf72-9e01ad752627 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1be5506d52574f8184f877d6aa9e8358'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:38.332884 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1315/2616] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:23:38 2022] DELETE /identity/v3/users/1be5506d52574f8184f877d6aa9e8358 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:38.539100 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85da121-2118-4105-9356-acc4450427ec service placement] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:38.553824 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85da121-2118-4105-9356-acc4450427ec service placement] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:38.617908 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba94943e-b174-42a1-b5bf-803d01ec8777 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:38.618530 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba94943e-b174-42a1-b5bf-803d01ec8777 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:38.618810 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba94943e-b174-42a1-b5bf-803d01ec8777 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:38.619089 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba94943e-b174-42a1-b5bf-803d01ec8777 admin admin] PATH_INFO: `/v3/projects/7ac42150994247a2a2f303085ac18420` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:38.620231 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba94943e-b174-42a1-b5bf-803d01ec8777 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7ac42150994247a2a2f303085ac18420)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:38.622951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba94943e-b174-42a1-b5bf-803d01ec8777 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:38.672063 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:38.682405 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-ba94943e-b174-42a1-b5bf-803d01ec8777 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7ac42150994247a2a2f303085ac18420', 'request_id': 'req-ba94943e-b174-42a1-b5bf-803d01ec8777'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:38.689933 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-ba94943e-b174-42a1-b5bf-803d01ec8777 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7ac42150994247a2a2f303085ac18420', 'request_id': 'req-ba94943e-b174-42a1-b5bf-803d01ec8777'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:38.695802 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-ba94943e-b174-42a1-b5bf-803d01ec8777 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7ac42150994247a2a2f303085ac18420', 'request_id': 'req-ba94943e-b174-42a1-b5bf-803d01ec8777'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:38.708977 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-ba94943e-b174-42a1-b5bf-803d01ec8777 admin admin] The token cache is being invalidate because project 7ac42150994247a2a2f303085ac18420 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:23:38.709263 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-ba94943e-b174-42a1-b5bf-803d01ec8777 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:38.709895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-ba94943e-b174-42a1-b5bf-803d01ec8777 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:38.711387 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1302/2617] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:23:38 2022] DELETE /identity/v3/projects/7ac42150994247a2a2f303085ac18420 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:38.716096 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33977334-2edb-422c-bf72-9e01ad752627 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:38.717658 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-33977334-2edb-422c-bf72-9e01ad752627 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:38.731798 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33977334-2edb-422c-bf72-9e01ad752627 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:38.751418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:38.792730 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76aaea9-5192-4749-b88d-59231c04ef25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:38.794045 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76aaea9-5192-4749-b88d-59231c04ef25 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:38.794314 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76aaea9-5192-4749-b88d-59231c04ef25 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:38.794589 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76aaea9-5192-4749-b88d-59231c04ef25 admin admin] PATH_INFO: `/v3/users/d413ebe892a94ec2b473eb71735c9941` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:38.795801 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76aaea9-5192-4749-b88d-59231c04ef25 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d413ebe892a94ec2b473eb71735c9941)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:38.797728 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76aaea9-5192-4749-b88d-59231c04ef25 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:38.847215 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-e76aaea9-5192-4749-b88d-59231c04ef25 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd413ebe892a94ec2b473eb71735c9941'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:38.847834 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-e76aaea9-5192-4749-b88d-59231c04ef25 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd413ebe892a94ec2b473eb71735c9941'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:38.848943 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-e76aaea9-5192-4749-b88d-59231c04ef25 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd413ebe892a94ec2b473eb71735c9941'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:38.856802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-e76aaea9-5192-4749-b88d-59231c04ef25 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd413ebe892a94ec2b473eb71735c9941'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:38.870123 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-e76aaea9-5192-4749-b88d-59231c04ef25 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd413ebe892a94ec2b473eb71735c9941'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:38.882407 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1316/2618] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:23:38 2022] DELETE /identity/v3/users/d413ebe892a94ec2b473eb71735c9941 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:39.060061 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba94943e-b174-42a1-b5bf-803d01ec8777 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:39.061519 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-ba94943e-b174-42a1-b5bf-803d01ec8777 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:39.074252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba94943e-b174-42a1-b5bf-803d01ec8777 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:39.132971 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0d03b8-298a-44b6-95c0-c379d0d81f2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:39.133592 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0d03b8-298a-44b6-95c0-c379d0d81f2d admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:39.133872 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0d03b8-298a-44b6-95c0-c379d0d81f2d admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:39.134144 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0d03b8-298a-44b6-95c0-c379d0d81f2d admin admin] PATH_INFO: `/v3/projects/cab13983d391495e8c069aa0b12b549d` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:39.135242 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0d03b8-298a-44b6-95c0-c379d0d81f2d admin admin] RBAC: Authorizing `identity:delete_project(project_id=cab13983d391495e8c069aa0b12b549d)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:39.137217 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0d03b8-298a-44b6-95c0-c379d0d81f2d admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:39.185832 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-ab0d03b8-298a-44b6-95c0-c379d0d81f2d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cab13983d391495e8c069aa0b12b549d', 'request_id': 'req-ab0d03b8-298a-44b6-95c0-c379d0d81f2d'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:39.192178 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-ab0d03b8-298a-44b6-95c0-c379d0d81f2d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cab13983d391495e8c069aa0b12b549d', 'request_id': 'req-ab0d03b8-298a-44b6-95c0-c379d0d81f2d'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:39.197669 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-ab0d03b8-298a-44b6-95c0-c379d0d81f2d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cab13983d391495e8c069aa0b12b549d', 'request_id': 'req-ab0d03b8-298a-44b6-95c0-c379d0d81f2d'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:39.210651 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-ab0d03b8-298a-44b6-95c0-c379d0d81f2d admin admin] The token cache is being invalidate because project cab13983d391495e8c069aa0b12b549d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:23:39.210943 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-ab0d03b8-298a-44b6-95c0-c379d0d81f2d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:39.211605 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-ab0d03b8-298a-44b6-95c0-c379d0d81f2d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:39.213109 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1303/2619] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:23:39 2022] DELETE /identity/v3/projects/cab13983d391495e8c069aa0b12b549d => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:39.219280 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8758a68-d90b-44f2-ab33-362eaafbeac3 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:39.219769 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8758a68-d90b-44f2-ab33-362eaafbeac3 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:39.219882 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8758a68-d90b-44f2-ab33-362eaafbeac3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:39.238992 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-a8758a68-d90b-44f2-ab33-362eaafbeac3 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:39.240233 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-a8758a68-d90b-44f2-ab33-362eaafbeac3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:39.301658 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1317/2620] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:39 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:23:39.310753 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7f46a8-aa4d-4c5c-907f-477dbd8207d0 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:39.311117 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7f46a8-aa4d-4c5c-907f-477dbd8207d0 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:39.311459 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7f46a8-aa4d-4c5c-907f-477dbd8207d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:39.331212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-1f7f46a8-aa4d-4c5c-907f-477dbd8207d0 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:39.332447 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-1f7f46a8-aa4d-4c5c-907f-477dbd8207d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:39.341289 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1304/2621] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:39 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:23:39.345520 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8758a68-d90b-44f2-ab33-362eaafbeac3 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:39.355402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8758a68-d90b-44f2-ab33-362eaafbeac3 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:39.360681 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523e9c3c-69f8-44ae-8c1a-31983d8d21f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:39.361289 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523e9c3c-69f8-44ae-8c1a-31983d8d21f8 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:39.361574 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523e9c3c-69f8-44ae-8c1a-31983d8d21f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:39.361854 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523e9c3c-69f8-44ae-8c1a-31983d8d21f8 admin admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:39.362510 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-523e9c3c-69f8-44ae-8c1a-31983d8d21f8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 16:23:39.362822 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-523e9c3c-69f8-44ae-8c1a-31983d8d21f8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:39.364463 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-523e9c3c-69f8-44ae-8c1a-31983d8d21f8 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:39.376534 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1318/2622] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 16:23:39 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 16:23:39.380886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f7f46a8-aa4d-4c5c-907f-477dbd8207d0 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:39.390881 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f7f46a8-aa4d-4c5c-907f-477dbd8207d0 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:39.396182 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912d8662-5d7c-411b-86e0-9b7b1053d50e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:39.396770 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912d8662-5d7c-411b-86e0-9b7b1053d50e admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:39.397054 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912d8662-5d7c-411b-86e0-9b7b1053d50e admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:39.397327 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912d8662-5d7c-411b-86e0-9b7b1053d50e admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:39.397964 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912d8662-5d7c-411b-86e0-9b7b1053d50e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:39.399883 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912d8662-5d7c-411b-86e0-9b7b1053d50e admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:39.427716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1305/2623] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:23:39 2022] POST /identity/v3/projects => generated 386 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:23:39.431831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523e9c3c-69f8-44ae-8c1a-31983d8d21f8 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:39.442098 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523e9c3c-69f8-44ae-8c1a-31983d8d21f8 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:39.469420 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:39.503563 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79428ed-2dfd-4f6c-89f9-2c5edcfb3dfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:39.504244 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79428ed-2dfd-4f6c-89f9-2c5edcfb3dfd admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:39.504510 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79428ed-2dfd-4f6c-89f9-2c5edcfb3dfd admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:39.504789 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79428ed-2dfd-4f6c-89f9-2c5edcfb3dfd admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:39.505429 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79428ed-2dfd-4f6c-89f9-2c5edcfb3dfd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:39.507359 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79428ed-2dfd-4f6c-89f9-2c5edcfb3dfd admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:39.530897 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1319/2624] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:23:39 2022] POST /identity/v3/users => generated 294 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 16:23:39.535233 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912d8662-5d7c-411b-86e0-9b7b1053d50e admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:39.545733 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912d8662-5d7c-411b-86e0-9b7b1053d50e admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:39.550436 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71a4949-3f5f-4668-aeee-bbe6661f8b82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:39.551030 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71a4949-3f5f-4668-aeee-bbe6661f8b82 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:39.551316 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71a4949-3f5f-4668-aeee-bbe6661f8b82 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:39.551590 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71a4949-3f5f-4668-aeee-bbe6661f8b82 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:39.552623 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b71a4949-3f5f-4668-aeee-bbe6661f8b82 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:39.554438 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b71a4949-3f5f-4668-aeee-bbe6661f8b82 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:39.564836 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1306/2625] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:23:39 2022] GET /identity/v3/roles => generated 1427 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:23:39.568825 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79428ed-2dfd-4f6c-89f9-2c5edcfb3dfd admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:39.579975 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79428ed-2dfd-4f6c-89f9-2c5edcfb3dfd admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:39.584854 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24f0e37-4ec1-4fec-a236-1e11ba0357d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:39.585450 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24f0e37-4ec1-4fec-a236-1e11ba0357d1 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:39.585729 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24f0e37-4ec1-4fec-a236-1e11ba0357d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:39.586001 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24f0e37-4ec1-4fec-a236-1e11ba0357d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:39.586639 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24f0e37-4ec1-4fec-a236-1e11ba0357d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:39.588283 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24f0e37-4ec1-4fec-a236-1e11ba0357d1 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:39.599579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1320/2626] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:23:39 2022] GET /identity/v3/roles => generated 1427 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:23:39.603491 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71a4949-3f5f-4668-aeee-bbe6661f8b82 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:39.623080 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71a4949-3f5f-4668-aeee-bbe6661f8b82 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:39.628463 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72ca3de-8093-4f43-afcc-1902abb12d36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:39.629122 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72ca3de-8093-4f43-afcc-1902abb12d36 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:39.629406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72ca3de-8093-4f43-afcc-1902abb12d36 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:39.629679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72ca3de-8093-4f43-afcc-1902abb12d36 admin admin] PATH_INFO: `/v3/projects/1978f4d7670043f794b6b75bb9d7bb66/users/7dc1abf6283a4adfa5dc019c74d3d567/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:39.643940 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:39.647648 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72ca3de-8093-4f43-afcc-1902abb12d36 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1978f4d7670043f794b6b75bb9d7bb66, user_id=7dc1abf6283a4adfa5dc019c74d3d567, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:39.650742 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72ca3de-8093-4f43-afcc-1902abb12d36 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:39.665682 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1307/2627] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:23:39 2022] PUT /identity/v3/projects/1978f4d7670043f794b6b75bb9d7bb66/users/7dc1abf6283a4adfa5dc019c74d3d567/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:39.676323 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c815e8e-0964-4b38-86fc-a5c6ae503a76 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:39.676791 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c815e8e-0964-4b38-86fc-a5c6ae503a76 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:39.676994 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c815e8e-0964-4b38-86fc-a5c6ae503a76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:39.694147 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72ca3de-8093-4f43-afcc-1902abb12d36 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:39.696000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-a72ca3de-8093-4f43-afcc-1902abb12d36 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:39.706988 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-1c815e8e-0964-4b38-86fc-a5c6ae503a76 None None] MFA Rules not processed for user `7dc1abf6283a4adfa5dc019c74d3d567`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:39.708227 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-1c815e8e-0964-4b38-86fc-a5c6ae503a76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:39.782155 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72ca3de-8093-4f43-afcc-1902abb12d36 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:39.789173 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1321/2628] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:39 2022] POST /identity/v3/auth/tokens => generated 3272 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:23:39.789563 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c7c197-8457-4787-bf60-903704f17776 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0ee0002a76c3471f819b37fef510ef99', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:39.790189 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c7c197-8457-4787-bf60-903704f17776 service placement] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:39.790475 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c7c197-8457-4787-bf60-903704f17776 service placement] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:39.790746 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c7c197-8457-4787-bf60-903704f17776 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:39.793684 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-39c7c197-8457-4787-bf60-903704f17776 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:39.797293 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c815e8e-0964-4b38-86fc-a5c6ae503a76 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:39.798814 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-1c815e8e-0964-4b38-86fc-a5c6ae503a76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:39.806537 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39c7c197-8457-4787-bf60-903704f17776 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:39.808191 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39c7c197-8457-4787-bf60-903704f17776 service placement] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:39.827657 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1308/2629] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:23:39 2022] GET /identity/v3/auth/tokens => generated 3318 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 16:23:39.872853 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c815e8e-0964-4b38-86fc-a5c6ae503a76 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:39.876656 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc4dadf-a370-4721-bd88-cb544185da17 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:39.877280 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc4dadf-a370-4721-bd88-cb544185da17 service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:39.877574 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc4dadf-a370-4721-bd88-cb544185da17 service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:39.877847 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc4dadf-a370-4721-bd88-cb544185da17 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:39.886430 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc4dadf-a370-4721-bd88-cb544185da17 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:39.888546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc4dadf-a370-4721-bd88-cb544185da17 service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:39.897632 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1322/2630] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:23:39 2022] GET /identity/v3/auth/tokens => generated 3272 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:23:39.918939 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c7c197-8457-4787-bf60-903704f17776 service placement] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:39.981074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c7c197-8457-4787-bf60-903704f17776 service placement] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:39.985716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd82d028-867c-41e6-94fe-ad97168b8ec1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:39.986337 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd82d028-867c-41e6-94fe-ad97168b8ec1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:39.986610 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd82d028-867c-41e6-94fe-ad97168b8ec1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:39.986877 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd82d028-867c-41e6-94fe-ad97168b8ec1 admin admin] PATH_INFO: `/v3/users/7dc1abf6283a4adfa5dc019c74d3d567` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:39.988030 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd82d028-867c-41e6-94fe-ad97168b8ec1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7dc1abf6283a4adfa5dc019c74d3d567)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:39.989929 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd82d028-867c-41e6-94fe-ad97168b8ec1 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:40.035333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-bd82d028-867c-41e6-94fe-ad97168b8ec1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7dc1abf6283a4adfa5dc019c74d3d567'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:40.036035 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-bd82d028-867c-41e6-94fe-ad97168b8ec1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7dc1abf6283a4adfa5dc019c74d3d567'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:40.042969 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-bd82d028-867c-41e6-94fe-ad97168b8ec1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7dc1abf6283a4adfa5dc019c74d3d567'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:40.055262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-bd82d028-867c-41e6-94fe-ad97168b8ec1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7dc1abf6283a4adfa5dc019c74d3d567'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:40.055939 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-bd82d028-867c-41e6-94fe-ad97168b8ec1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7dc1abf6283a4adfa5dc019c74d3d567'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:40.066083 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:40.070500 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1309/2631] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:23:39 2022] DELETE /identity/v3/users/7dc1abf6283a4adfa5dc019c74d3d567 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:40.075843 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efecc72-f372-48ac-9f41-d735747b4a5c None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:40.076214 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efecc72-f372-48ac-9f41-d735747b4a5c None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:40.076624 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efecc72-f372-48ac-9f41-d735747b4a5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:40.097706 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-4efecc72-f372-48ac-9f41-d735747b4a5c None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:40.098923 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-4efecc72-f372-48ac-9f41-d735747b4a5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:40.139732 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:40.177448 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1323/2632] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:40 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:23:40.187927 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd82d028-867c-41e6-94fe-ad97168b8ec1 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:40.189337 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-bd82d028-867c-41e6-94fe-ad97168b8ec1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:40.257596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd82d028-867c-41e6-94fe-ad97168b8ec1 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:40.261257 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50679fdb-2563-45c3-b2fb-85adc48220cc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:40.261973 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50679fdb-2563-45c3-b2fb-85adc48220cc service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:40.262321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50679fdb-2563-45c3-b2fb-85adc48220cc service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:40.262655 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50679fdb-2563-45c3-b2fb-85adc48220cc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:40.271884 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50679fdb-2563-45c3-b2fb-85adc48220cc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:40.273355 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50679fdb-2563-45c3-b2fb-85adc48220cc service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:40.283979 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1310/2633] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:23:40 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:23:40.578302 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efecc72-f372-48ac-9f41-d735747b4a5c None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:40.579856 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-4efecc72-f372-48ac-9f41-d735747b4a5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:40.594622 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efecc72-f372-48ac-9f41-d735747b4a5c None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:40.600311 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0655df41-c905-4a3b-94a4-7adecd3880fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:40.601019 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0655df41-c905-4a3b-94a4-7adecd3880fd admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:40.601224 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0655df41-c905-4a3b-94a4-7adecd3880fd admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:40.601499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0655df41-c905-4a3b-94a4-7adecd3880fd admin admin] PATH_INFO: `/v3/projects/1978f4d7670043f794b6b75bb9d7bb66` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:40.602630 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0655df41-c905-4a3b-94a4-7adecd3880fd admin admin] RBAC: Authorizing `identity:delete_project(project_id=1978f4d7670043f794b6b75bb9d7bb66)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:40.604652 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0655df41-c905-4a3b-94a4-7adecd3880fd admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:40.651135 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0655df41-c905-4a3b-94a4-7adecd3880fd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1978f4d7670043f794b6b75bb9d7bb66', 'request_id': 'req-0655df41-c905-4a3b-94a4-7adecd3880fd'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:40.658578 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0655df41-c905-4a3b-94a4-7adecd3880fd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1978f4d7670043f794b6b75bb9d7bb66', 'request_id': 'req-0655df41-c905-4a3b-94a4-7adecd3880fd'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:40.664581 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0655df41-c905-4a3b-94a4-7adecd3880fd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1978f4d7670043f794b6b75bb9d7bb66', 'request_id': 'req-0655df41-c905-4a3b-94a4-7adecd3880fd'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:40.678244 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0655df41-c905-4a3b-94a4-7adecd3880fd admin admin] The token cache is being invalidate because project 1978f4d7670043f794b6b75bb9d7bb66 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:23:40.678534 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0655df41-c905-4a3b-94a4-7adecd3880fd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:40.679167 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0655df41-c905-4a3b-94a4-7adecd3880fd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:40.680774 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1324/2634] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:23:40 2022] DELETE /identity/v3/projects/1978f4d7670043f794b6b75bb9d7bb66 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:40.685544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc828758-1891-4507-80d0-e221a693238c None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:40.685836 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc828758-1891-4507-80d0-e221a693238c None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:40.686109 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc828758-1891-4507-80d0-e221a693238c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:40.705690 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-dc828758-1891-4507-80d0-e221a693238c None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:40.706880 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-dc828758-1891-4507-80d0-e221a693238c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:40.767048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1311/2635] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:40 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:23:40.774921 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbb2092-2316-4ef9-8718-c8067194d865 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:40.775193 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbb2092-2316-4ef9-8718-c8067194d865 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:40.775469 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbb2092-2316-4ef9-8718-c8067194d865 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:40.795732 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-6cbb2092-2316-4ef9-8718-c8067194d865 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:40.796868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-6cbb2092-2316-4ef9-8718-c8067194d865 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:40.804759 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1325/2636] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:40 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:23:40.809702 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc828758-1891-4507-80d0-e221a693238c None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:40.819691 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc828758-1891-4507-80d0-e221a693238c None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:40.824379 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2671743-4372-43ba-9aed-414bf392431d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:40.824991 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2671743-4372-43ba-9aed-414bf392431d admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:40.825269 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2671743-4372-43ba-9aed-414bf392431d admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:40.825567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2671743-4372-43ba-9aed-414bf392431d admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:40.826221 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2671743-4372-43ba-9aed-414bf392431d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 16:23:40.826530 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2671743-4372-43ba-9aed-414bf392431d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:40.828275 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2671743-4372-43ba-9aed-414bf392431d admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:40.840652 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1312/2637] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 16:23:40 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 16:23:40.845070 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cbb2092-2316-4ef9-8718-c8067194d865 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:40.854539 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cbb2092-2316-4ef9-8718-c8067194d865 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:40.860662 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82951770-1a24-4f73-8753-345c6f430dff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:40.861302 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82951770-1a24-4f73-8753-345c6f430dff admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:40.861579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82951770-1a24-4f73-8753-345c6f430dff admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:40.862345 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82951770-1a24-4f73-8753-345c6f430dff admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:40.862513 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82951770-1a24-4f73-8753-345c6f430dff admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:40.864480 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82951770-1a24-4f73-8753-345c6f430dff admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:40.893942 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1326/2638] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:23:40 2022] POST /identity/v3/projects => generated 378 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:23:40.898840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2671743-4372-43ba-9aed-414bf392431d admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:40.925668 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2671743-4372-43ba-9aed-414bf392431d admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:41.002198 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96377a7b-cff2-4073-9747-8cfc84d12d64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:41.002792 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96377a7b-cff2-4073-9747-8cfc84d12d64 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:41.003075 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96377a7b-cff2-4073-9747-8cfc84d12d64 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:41.003353 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96377a7b-cff2-4073-9747-8cfc84d12d64 admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:41.004025 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96377a7b-cff2-4073-9747-8cfc84d12d64 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:41.005952 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96377a7b-cff2-4073-9747-8cfc84d12d64 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:41.030038 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1313/2639] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:23:40 2022] POST /identity/v3/users => generated 290 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:23:41.034412 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82951770-1a24-4f73-8753-345c6f430dff admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:41.046214 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82951770-1a24-4f73-8753-345c6f430dff admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:41.050938 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0106c677-dc24-48d0-b5c6-70521ae99bb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:41.051546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0106c677-dc24-48d0-b5c6-70521ae99bb6 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:41.051856 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0106c677-dc24-48d0-b5c6-70521ae99bb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:41.052133 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0106c677-dc24-48d0-b5c6-70521ae99bb6 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:41.052842 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0106c677-dc24-48d0-b5c6-70521ae99bb6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:41.054411 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0106c677-dc24-48d0-b5c6-70521ae99bb6 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:41.074193 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1327/2640] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:23:41 2022] GET /identity/v3/roles => generated 1427 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:23:41.078325 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96377a7b-cff2-4073-9747-8cfc84d12d64 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:41.088571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96377a7b-cff2-4073-9747-8cfc84d12d64 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:41.094429 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d7450c-9951-4b23-a086-901b836696ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:41.095023 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d7450c-9951-4b23-a086-901b836696ea admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:41.095334 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d7450c-9951-4b23-a086-901b836696ea admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:41.095609 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d7450c-9951-4b23-a086-901b836696ea admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:41.096276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d7450c-9951-4b23-a086-901b836696ea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:41.097891 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d7450c-9951-4b23-a086-901b836696ea admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:41.109112 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1314/2641] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:23:41 2022] GET /identity/v3/roles => generated 1427 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:23:41.113194 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0106c677-dc24-48d0-b5c6-70521ae99bb6 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:41.124090 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0106c677-dc24-48d0-b5c6-70521ae99bb6 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:41.128762 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4229bac4-3d06-47a1-a308-275e7fd6544a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:41.129419 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4229bac4-3d06-47a1-a308-275e7fd6544a admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:41.129698 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4229bac4-3d06-47a1-a308-275e7fd6544a admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:41.129972 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4229bac4-3d06-47a1-a308-275e7fd6544a admin admin] PATH_INFO: `/v3/projects/e3593174124f432fbcc8e026c014ca10/users/c0d32d5a4e4045cf97019a66a7d0c0d3/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:41.145616 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4229bac4-3d06-47a1-a308-275e7fd6544a admin admin] RBAC: Authorizing `identity:create_grant(project_id=e3593174124f432fbcc8e026c014ca10, user_id=c0d32d5a4e4045cf97019a66a7d0c0d3, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:41.148290 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4229bac4-3d06-47a1-a308-275e7fd6544a admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:41.161823 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1328/2642] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:23:41 2022] PUT /identity/v3/projects/e3593174124f432fbcc8e026c014ca10/users/c0d32d5a4e4045cf97019a66a7d0c0d3/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:41.172993 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3d7e57-78aa-401f-b9d8-641eb66b453b None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:41.173201 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3d7e57-78aa-401f-b9d8-641eb66b453b None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:41.173477 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3d7e57-78aa-401f-b9d8-641eb66b453b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:41.193643 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:41.204441 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-6c3d7e57-78aa-401f-b9d8-641eb66b453b None None] MFA Rules not processed for user `c0d32d5a4e4045cf97019a66a7d0c0d3`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:41.206012 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-6c3d7e57-78aa-401f-b9d8-641eb66b453b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:41.271749 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1315/2643] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:41 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:23:41.276780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4229bac4-3d06-47a1-a308-275e7fd6544a admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:41.339399 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4229bac4-3d06-47a1-a308-275e7fd6544a admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:41.344305 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef10f805-b73e-4bd3-a562-fa05a3a7d8c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:41.344936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef10f805-b73e-4bd3-a562-fa05a3a7d8c1 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:41.345210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef10f805-b73e-4bd3-a562-fa05a3a7d8c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:41.345478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef10f805-b73e-4bd3-a562-fa05a3a7d8c1 admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:41.346101 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef10f805-b73e-4bd3-a562-fa05a3a7d8c1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:41.348038 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef10f805-b73e-4bd3-a562-fa05a3a7d8c1 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:41.352540 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:41.376727 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1329/2644] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:23:41 2022] POST /identity/v3/projects => generated 378 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:23:41.380777 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3d7e57-78aa-401f-b9d8-641eb66b453b None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:41.392138 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3d7e57-78aa-401f-b9d8-641eb66b453b None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:41.447842 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd373bc-3051-4b79-a89e-d77deecb9bef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:41.448352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd373bc-3051-4b79-a89e-d77deecb9bef admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:41.448626 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd373bc-3051-4b79-a89e-d77deecb9bef admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:41.448895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd373bc-3051-4b79-a89e-d77deecb9bef admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:41.449540 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd373bc-3051-4b79-a89e-d77deecb9bef admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:41.451441 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd373bc-3051-4b79-a89e-d77deecb9bef admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:41.473653 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1316/2645] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:23:41 2022] POST /identity/v3/users => generated 290 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:23:41.477695 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef10f805-b73e-4bd3-a562-fa05a3a7d8c1 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:41.489453 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef10f805-b73e-4bd3-a562-fa05a3a7d8c1 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:41.494551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fbf2b2e-133d-4a64-8694-0ff21b8b2c47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:41.495160 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fbf2b2e-133d-4a64-8694-0ff21b8b2c47 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:41.495446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fbf2b2e-133d-4a64-8694-0ff21b8b2c47 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:41.495761 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fbf2b2e-133d-4a64-8694-0ff21b8b2c47 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:41.497893 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fbf2b2e-133d-4a64-8694-0ff21b8b2c47 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:41.499503 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fbf2b2e-133d-4a64-8694-0ff21b8b2c47 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:41.512697 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1330/2646] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:23:41 2022] GET /identity/v3/roles => generated 1427 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 16:23:41.514952 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd373bc-3051-4b79-a89e-d77deecb9bef admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:41.525081 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd373bc-3051-4b79-a89e-d77deecb9bef admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:41.534534 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb4aa42-2533-4f73-b121-b30cbb06e061 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:41.535141 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb4aa42-2533-4f73-b121-b30cbb06e061 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:41.535420 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb4aa42-2533-4f73-b121-b30cbb06e061 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:41.535720 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb4aa42-2533-4f73-b121-b30cbb06e061 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:41.536366 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb4aa42-2533-4f73-b121-b30cbb06e061 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:41.538082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb4aa42-2533-4f73-b121-b30cbb06e061 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:41.549282 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1317/2647] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:23:41 2022] GET /identity/v3/roles => generated 1427 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:23:41.553204 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fbf2b2e-133d-4a64-8694-0ff21b8b2c47 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:41.568632 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fbf2b2e-133d-4a64-8694-0ff21b8b2c47 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:41.573328 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd744e98-9bf6-43f4-914d-b579a9a9221c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:41.573992 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd744e98-9bf6-43f4-914d-b579a9a9221c admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:41.574279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd744e98-9bf6-43f4-914d-b579a9a9221c admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:41.575042 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd744e98-9bf6-43f4-914d-b579a9a9221c admin admin] PATH_INFO: `/v3/projects/bfb363e22d92402ea36bc01526546eac/users/8cd8822f60a749a2972c54fb10a102a3/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:41.590986 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd744e98-9bf6-43f4-914d-b579a9a9221c admin admin] RBAC: Authorizing `identity:create_grant(project_id=bfb363e22d92402ea36bc01526546eac, user_id=8cd8822f60a749a2972c54fb10a102a3, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:41.593709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd744e98-9bf6-43f4-914d-b579a9a9221c admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:41.612805 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1331/2648] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:23:41 2022] PUT /identity/v3/projects/bfb363e22d92402ea36bc01526546eac/users/8cd8822f60a749a2972c54fb10a102a3/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:41.623600 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c58c4d-c0c6-421c-919c-cb76629d8ff0 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:41.623916 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c58c4d-c0c6-421c-919c-cb76629d8ff0 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:41.624194 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c58c4d-c0c6-421c-919c-cb76629d8ff0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:41.653014 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-63c58c4d-c0c6-421c-919c-cb76629d8ff0 None None] MFA Rules not processed for user `8cd8822f60a749a2972c54fb10a102a3`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:41.654220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-63c58c4d-c0c6-421c-919c-cb76629d8ff0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:41.722591 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1318/2649] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:41 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:23:41.727094 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd744e98-9bf6-43f4-914d-b579a9a9221c admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:41.800246 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd744e98-9bf6-43f4-914d-b579a9a9221c admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:41.804988 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390d53e1-d3bb-4153-ae77-e676185caba3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:41.805592 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390d53e1-d3bb-4153-ae77-e676185caba3 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:41.805869 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390d53e1-d3bb-4153-ae77-e676185caba3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:41.806147 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390d53e1-d3bb-4153-ae77-e676185caba3 admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:41.806767 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390d53e1-d3bb-4153-ae77-e676185caba3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:41.809666 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390d53e1-d3bb-4153-ae77-e676185caba3 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:41.837207 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1332/2650] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:23:41 2022] POST /identity/v3/projects => generated 376 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:23:41.842060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c58c4d-c0c6-421c-919c-cb76629d8ff0 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:41.852906 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c58c4d-c0c6-421c-919c-cb76629d8ff0 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:41.856500 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:41.914379 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7936f597-16d4-4da5-8233-8934aae469b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:41.914982 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7936f597-16d4-4da5-8233-8934aae469b6 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:41.915261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7936f597-16d4-4da5-8233-8934aae469b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:41.915527 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7936f597-16d4-4da5-8233-8934aae469b6 admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:41.918060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7936f597-16d4-4da5-8233-8934aae469b6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:41.918261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7936f597-16d4-4da5-8233-8934aae469b6 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:41.946141 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1319/2651] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:23:41 2022] POST /identity/v3/users => generated 295 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 16:23:41.950809 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390d53e1-d3bb-4153-ae77-e676185caba3 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:41.963230 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390d53e1-d3bb-4153-ae77-e676185caba3 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:41.969368 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805bdf63-1ab7-483b-ba41-2c03f5fbf5aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:41.970221 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805bdf63-1ab7-483b-ba41-2c03f5fbf5aa admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:41.970551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805bdf63-1ab7-483b-ba41-2c03f5fbf5aa admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:41.971149 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805bdf63-1ab7-483b-ba41-2c03f5fbf5aa admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:41.971699 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805bdf63-1ab7-483b-ba41-2c03f5fbf5aa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:41.974207 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805bdf63-1ab7-483b-ba41-2c03f5fbf5aa admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:41.987155 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1333/2652] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:23:41 2022] GET /identity/v3/roles => generated 1427 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:23:41.991727 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7936f597-16d4-4da5-8233-8934aae469b6 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:42.003908 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7936f597-16d4-4da5-8233-8934aae469b6 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:42.009547 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1e112d-4032-4281-b617-8d2132263c44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:42.010714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1e112d-4032-4281-b617-8d2132263c44 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:42.010848 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1e112d-4032-4281-b617-8d2132263c44 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:42.011178 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1e112d-4032-4281-b617-8d2132263c44 admin admin] PATH_INFO: `/v3/projects/e06adf1f4c764d8897db902995eab4ba/users/0e77c0e2e9be4ba4b383cd994a533dc9/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:42.033345 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1e112d-4032-4281-b617-8d2132263c44 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e06adf1f4c764d8897db902995eab4ba, user_id=0e77c0e2e9be4ba4b383cd994a533dc9, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:42.037289 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1e112d-4032-4281-b617-8d2132263c44 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:42.055859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1320/2653] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:23:41 2022] PUT /identity/v3/projects/e06adf1f4c764d8897db902995eab4ba/users/0e77c0e2e9be4ba4b383cd994a533dc9/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:42.061210 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805bdf63-1ab7-483b-ba41-2c03f5fbf5aa admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:42.074484 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805bdf63-1ab7-483b-ba41-2c03f5fbf5aa admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:42.164512 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a05016-ff1f-452d-a491-51d52c7b57b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:42.165826 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a05016-ff1f-452d-a491-51d52c7b57b5 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:42.165966 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a05016-ff1f-452d-a491-51d52c7b57b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:42.166423 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a05016-ff1f-452d-a491-51d52c7b57b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:42.167574 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a05016-ff1f-452d-a491-51d52c7b57b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:42.172099 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a05016-ff1f-452d-a491-51d52c7b57b5 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:42.188433 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1334/2654] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:23:42 2022] GET /identity/v3/roles => generated 1427 bytes in 127 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:23:42.192264 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1e112d-4032-4281-b617-8d2132263c44 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:42.206740 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1e112d-4032-4281-b617-8d2132263c44 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:42.217225 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f30ea1f-8152-4246-8a3f-c8e6b3afc8b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:42.219134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f30ea1f-8152-4246-8a3f-c8e6b3afc8b6 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:42.219826 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f30ea1f-8152-4246-8a3f-c8e6b3afc8b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:42.220199 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f30ea1f-8152-4246-8a3f-c8e6b3afc8b6 admin admin] PATH_INFO: `/v3/projects/e06adf1f4c764d8897db902995eab4ba/users/0e77c0e2e9be4ba4b383cd994a533dc9/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:42.222598 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f30ea1f-8152-4246-8a3f-c8e6b3afc8b6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e06adf1f4c764d8897db902995eab4ba, user_id=0e77c0e2e9be4ba4b383cd994a533dc9, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:42.226900 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f30ea1f-8152-4246-8a3f-c8e6b3afc8b6 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:42.265330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1321/2655] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:23:42 2022] PUT /identity/v3/projects/e06adf1f4c764d8897db902995eab4ba/users/0e77c0e2e9be4ba4b383cd994a533dc9/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:23:42.273279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ab295d-f484-4624-932c-c2f24840b063 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:42.273279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ab295d-f484-4624-932c-c2f24840b063 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:42.273279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ab295d-f484-4624-932c-c2f24840b063 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:42.310499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-29ab295d-f484-4624-932c-c2f24840b063 None None] MFA Rules not processed for user `0e77c0e2e9be4ba4b383cd994a533dc9`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:42.311817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-29ab295d-f484-4624-932c-c2f24840b063 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:42.320626 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:42.384393 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1335/2656] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:42 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:23:42.396878 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f30ea1f-8152-4246-8a3f-c8e6b3afc8b6 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:42.398592 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-3f30ea1f-8152-4246-8a3f-c8e6b3afc8b6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:42.457344 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f30ea1f-8152-4246-8a3f-c8e6b3afc8b6 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:42.461757 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1196b53-6034-4b41-9587-8b132767d61f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:42.462364 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1196b53-6034-4b41-9587-8b132767d61f service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:42.462642 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1196b53-6034-4b41-9587-8b132767d61f service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:42.462915 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1196b53-6034-4b41-9587-8b132767d61f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:42.519546 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1196b53-6034-4b41-9587-8b132767d61f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:42.520926 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1196b53-6034-4b41-9587-8b132767d61f service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:42.530548 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1322/2657] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:23:42 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:23:43.203441 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ab295d-f484-4624-932c-c2f24840b063 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:43.204918 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-29ab295d-f484-4624-932c-c2f24840b063 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:43.299114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ab295d-f484-4624-932c-c2f24840b063 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:43.304369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f988ca-208a-4e7f-8516-5cdc357694d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:43.305191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f988ca-208a-4e7f-8516-5cdc357694d0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:43.305505 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f988ca-208a-4e7f-8516-5cdc357694d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:43.305909 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f988ca-208a-4e7f-8516-5cdc357694d0 admin admin] PATH_INFO: `/v3/users/94c85145df4f4c43a5f12f873d273543` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:43.307431 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f988ca-208a-4e7f-8516-5cdc357694d0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=94c85145df4f4c43a5f12f873d273543)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:43.310337 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f988ca-208a-4e7f-8516-5cdc357694d0 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:43.372162 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-71f988ca-208a-4e7f-8516-5cdc357694d0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '94c85145df4f4c43a5f12f873d273543'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:43.372966 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-71f988ca-208a-4e7f-8516-5cdc357694d0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '94c85145df4f4c43a5f12f873d273543'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:43.373579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-71f988ca-208a-4e7f-8516-5cdc357694d0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '94c85145df4f4c43a5f12f873d273543'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:43.381556 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-71f988ca-208a-4e7f-8516-5cdc357694d0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '94c85145df4f4c43a5f12f873d273543'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:43.396828 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-71f988ca-208a-4e7f-8516-5cdc357694d0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '94c85145df4f4c43a5f12f873d273543'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:43.409360 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1336/2658] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:23:43 2022] DELETE /identity/v3/users/94c85145df4f4c43a5f12f873d273543 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:43.413979 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760ba2c3-0fc0-46b9-bd51-a5874ec0deaa None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:43.414495 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760ba2c3-0fc0-46b9-bd51-a5874ec0deaa None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:43.414586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760ba2c3-0fc0-46b9-bd51-a5874ec0deaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:43.434613 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-760ba2c3-0fc0-46b9-bd51-a5874ec0deaa None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:43.435860 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-760ba2c3-0fc0-46b9-bd51-a5874ec0deaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:43.444042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:43.504945 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1323/2659] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:43 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:23:43.515768 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f988ca-208a-4e7f-8516-5cdc357694d0 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:43.517226 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-71f988ca-208a-4e7f-8516-5cdc357694d0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:43.547089 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:43.580936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f988ca-208a-4e7f-8516-5cdc357694d0 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:43.584788 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabed41f-3e34-44ae-8e25-ed3309b8d55e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:43.585488 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabed41f-3e34-44ae-8e25-ed3309b8d55e service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:43.585875 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabed41f-3e34-44ae-8e25-ed3309b8d55e service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:43.585999 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabed41f-3e34-44ae-8e25-ed3309b8d55e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:43.596414 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dabed41f-3e34-44ae-8e25-ed3309b8d55e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:43.597862 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dabed41f-3e34-44ae-8e25-ed3309b8d55e service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:43.607584 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1337/2660] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:23:43 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:23:43.929729 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760ba2c3-0fc0-46b9-bd51-a5874ec0deaa None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:43.931169 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-760ba2c3-0fc0-46b9-bd51-a5874ec0deaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:43.956794 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760ba2c3-0fc0-46b9-bd51-a5874ec0deaa None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:43.979049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf112a65-696b-49b1-9b1a-716366a1964a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:43.981559 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf112a65-696b-49b1-9b1a-716366a1964a admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:43.983675 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf112a65-696b-49b1-9b1a-716366a1964a admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:43.983675 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf112a65-696b-49b1-9b1a-716366a1964a admin admin] PATH_INFO: `/v3/projects/f53ca6f714944d39b0be393a31581e68` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:43.987962 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf112a65-696b-49b1-9b1a-716366a1964a admin admin] RBAC: Authorizing `identity:delete_project(project_id=f53ca6f714944d39b0be393a31581e68)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:43.989988 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf112a65-696b-49b1-9b1a-716366a1964a admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:44.057531 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-bf112a65-696b-49b1-9b1a-716366a1964a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f53ca6f714944d39b0be393a31581e68', 'request_id': 'req-bf112a65-696b-49b1-9b1a-716366a1964a'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:44.065840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-bf112a65-696b-49b1-9b1a-716366a1964a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f53ca6f714944d39b0be393a31581e68', 'request_id': 'req-bf112a65-696b-49b1-9b1a-716366a1964a'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:44.074872 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-bf112a65-696b-49b1-9b1a-716366a1964a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f53ca6f714944d39b0be393a31581e68', 'request_id': 'req-bf112a65-696b-49b1-9b1a-716366a1964a'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:44.090571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-bf112a65-696b-49b1-9b1a-716366a1964a admin admin] The token cache is being invalidate because project f53ca6f714944d39b0be393a31581e68 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:23:44.091125 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-bf112a65-696b-49b1-9b1a-716366a1964a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:44.092023 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-bf112a65-696b-49b1-9b1a-716366a1964a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:44.093837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1324/2661] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:23:43 2022] DELETE /identity/v3/projects/f53ca6f714944d39b0be393a31581e68 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:44.098003 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabed41f-3e34-44ae-8e25-ed3309b8d55e service neutron] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:44.108845 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf112a65-696b-49b1-9b1a-716366a1964a admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:44.109837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabed41f-3e34-44ae-8e25-ed3309b8d55e service neutron] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:44.111700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-bf112a65-696b-49b1-9b1a-716366a1964a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:44.183298 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e61d54c-42c5-4bb0-b30a-0f3473645777 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:44.185548 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e61d54c-42c5-4bb0-b30a-0f3473645777 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:44.186163 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e61d54c-42c5-4bb0-b30a-0f3473645777 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:44.186607 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e61d54c-42c5-4bb0-b30a-0f3473645777 admin admin] PATH_INFO: `/v3/users/be56c66194cf43c3806b33a4ffaed685` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:44.188174 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e61d54c-42c5-4bb0-b30a-0f3473645777 admin admin] RBAC: Authorizing `identity:delete_user(user_id=be56c66194cf43c3806b33a4ffaed685)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:44.189665 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf112a65-696b-49b1-9b1a-716366a1964a admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:44.190492 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e61d54c-42c5-4bb0-b30a-0f3473645777 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:44.194200 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ebdb20-0236-4021-bf89-84ad23c244a6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:44.194831 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ebdb20-0236-4021-bf89-84ad23c244a6 service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:44.195290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ebdb20-0236-4021-bf89-84ad23c244a6 service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:44.195731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ebdb20-0236-4021-bf89-84ad23c244a6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:44.197829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-08ebdb20-0236-4021-bf89-84ad23c244a6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:44.260075 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-5e61d54c-42c5-4bb0-b30a-0f3473645777 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'be56c66194cf43c3806b33a4ffaed685'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:44.260790 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-5e61d54c-42c5-4bb0-b30a-0f3473645777 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'be56c66194cf43c3806b33a4ffaed685'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:44.261746 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-5e61d54c-42c5-4bb0-b30a-0f3473645777 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'be56c66194cf43c3806b33a4ffaed685'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:44.265924 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:44.272982 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-5e61d54c-42c5-4bb0-b30a-0f3473645777 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'be56c66194cf43c3806b33a4ffaed685'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:44.284362 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ebdb20-0236-4021-bf89-84ad23c244a6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:44.287450 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ebdb20-0236-4021-bf89-84ad23c244a6 service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:44.296112 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-5e61d54c-42c5-4bb0-b30a-0f3473645777 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'be56c66194cf43c3806b33a4ffaed685'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:44.300406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1325/2662] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:23:44 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 192 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 16:23:44.310792 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1338/2663] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:23:44 2022] DELETE /identity/v3/users/be56c66194cf43c3806b33a4ffaed685 => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:44.720552 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ebdb20-0236-4021-bf89-84ad23c244a6 service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:44.722461 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-08ebdb20-0236-4021-bf89-84ad23c244a6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:44.765900 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ebdb20-0236-4021-bf89-84ad23c244a6 service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:44.868157 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b34a045-0bef-41fa-8fb2-53a57a3e7853 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:44.869902 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b34a045-0bef-41fa-8fb2-53a57a3e7853 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:44.870193 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b34a045-0bef-41fa-8fb2-53a57a3e7853 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:44.871725 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b34a045-0bef-41fa-8fb2-53a57a3e7853 admin admin] PATH_INFO: `/v3/projects/c3c6acb87e03462da9ae56a61e65cafb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:44.874713 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b34a045-0bef-41fa-8fb2-53a57a3e7853 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c3c6acb87e03462da9ae56a61e65cafb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:44.876829 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b34a045-0bef-41fa-8fb2-53a57a3e7853 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:44.941306 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-6b34a045-0bef-41fa-8fb2-53a57a3e7853 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c3c6acb87e03462da9ae56a61e65cafb', 'request_id': 'req-6b34a045-0bef-41fa-8fb2-53a57a3e7853'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:44.949701 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-6b34a045-0bef-41fa-8fb2-53a57a3e7853 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c3c6acb87e03462da9ae56a61e65cafb', 'request_id': 'req-6b34a045-0bef-41fa-8fb2-53a57a3e7853'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:44.959125 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-6b34a045-0bef-41fa-8fb2-53a57a3e7853 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c3c6acb87e03462da9ae56a61e65cafb', 'request_id': 'req-6b34a045-0bef-41fa-8fb2-53a57a3e7853'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:44.987718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-6b34a045-0bef-41fa-8fb2-53a57a3e7853 admin admin] The token cache is being invalidate because project c3c6acb87e03462da9ae56a61e65cafb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:23:44.987718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-6b34a045-0bef-41fa-8fb2-53a57a3e7853 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:44.987718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-6b34a045-0bef-41fa-8fb2-53a57a3e7853 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:44.987718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1326/2664] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:23:44 2022] DELETE /identity/v3/projects/c3c6acb87e03462da9ae56a61e65cafb => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:44.997912 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45d4385-bbaf-4da4-ac08-a6dc1b1012b4 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:44.998364 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45d4385-bbaf-4da4-ac08-a6dc1b1012b4 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:44.998707 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45d4385-bbaf-4da4-ac08-a6dc1b1012b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:45.024659 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-e45d4385-bbaf-4da4-ac08-a6dc1b1012b4 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:45.026935 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-e45d4385-bbaf-4da4-ac08-a6dc1b1012b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:45.079325 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:45.096584 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b34a045-0bef-41fa-8fb2-53a57a3e7853 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:45.098114 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-6b34a045-0bef-41fa-8fb2-53a57a3e7853 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:45.101219 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1339/2665] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:44 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:23:45.109352 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8914da8-5172-41a5-a01d-263a5cca3df2 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:45.109742 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8914da8-5172-41a5-a01d-263a5cca3df2 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:45.110120 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8914da8-5172-41a5-a01d-263a5cca3df2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:45.131347 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-b8914da8-5172-41a5-a01d-263a5cca3df2 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:45.132739 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-b8914da8-5172-41a5-a01d-263a5cca3df2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:45.142986 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1340/2666] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:45 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 16:23:45.147128 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8914da8-5172-41a5-a01d-263a5cca3df2 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:45.157299 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8914da8-5172-41a5-a01d-263a5cca3df2 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:45.162613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63e17ec-6263-411c-90a6-f2b542131e83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:45.163359 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63e17ec-6263-411c-90a6-f2b542131e83 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:45.163798 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63e17ec-6263-411c-90a6-f2b542131e83 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:45.164581 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63e17ec-6263-411c-90a6-f2b542131e83 admin admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:45.165335 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a63e17ec-6263-411c-90a6-f2b542131e83 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 16:23:45.165857 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a63e17ec-6263-411c-90a6-f2b542131e83 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:45.167442 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a63e17ec-6263-411c-90a6-f2b542131e83 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:45.176011 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b34a045-0bef-41fa-8fb2-53a57a3e7853 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:45.180406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe8eae5-3069-4d75-97ca-d74a778bba62 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:45.182136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1341/2667] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 16:23:45 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 16:23:45.182466 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe8eae5-3069-4d75-97ca-d74a778bba62 service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:45.183085 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe8eae5-3069-4d75-97ca-d74a778bba62 service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:45.184465 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe8eae5-3069-4d75-97ca-d74a778bba62 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:45.187084 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-0fe8eae5-3069-4d75-97ca-d74a778bba62 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:45.187259 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63e17ec-6263-411c-90a6-f2b542131e83 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:45.198010 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63e17ec-6263-411c-90a6-f2b542131e83 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:45.204418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3708383e-4ac6-4f3e-93f6-541f19e6dd19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:45.205051 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3708383e-4ac6-4f3e-93f6-541f19e6dd19 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:45.205334 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3708383e-4ac6-4f3e-93f6-541f19e6dd19 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:45.205609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3708383e-4ac6-4f3e-93f6-541f19e6dd19 admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:45.206266 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3708383e-4ac6-4f3e-93f6-541f19e6dd19 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:45.218842 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3708383e-4ac6-4f3e-93f6-541f19e6dd19 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:45.233421 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:45.263724 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1342/2668] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:23:45 2022] POST /identity/v3/projects => generated 394 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 16:23:45.266388 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3708383e-4ac6-4f3e-93f6-541f19e6dd19 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:45.285369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fe8eae5-3069-4d75-97ca-d74a778bba62 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:45.286917 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fe8eae5-3069-4d75-97ca-d74a778bba62 service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:45.305412 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1327/2669] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:23:45 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 205 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:23:45.373480 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3708383e-4ac6-4f3e-93f6-541f19e6dd19 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:45.380503 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ce6200-b5bd-48d1-81b1-251583dd30dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:45.381428 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ce6200-b5bd-48d1-81b1-251583dd30dc admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:45.381989 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ce6200-b5bd-48d1-81b1-251583dd30dc admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:45.382552 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ce6200-b5bd-48d1-81b1-251583dd30dc admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:45.383346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ce6200-b5bd-48d1-81b1-251583dd30dc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:45.385348 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ce6200-b5bd-48d1-81b1-251583dd30dc admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:45.420589 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1343/2670] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:23:45 2022] POST /identity/v3/users => generated 298 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:23:45.426261 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe8eae5-3069-4d75-97ca-d74a778bba62 service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:45.445560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe8eae5-3069-4d75-97ca-d74a778bba62 service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:45.453063 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb0bfbe-5618-49c4-8e5c-2e2bedf3ffcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:45.453766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb0bfbe-5618-49c4-8e5c-2e2bedf3ffcb admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:45.454106 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb0bfbe-5618-49c4-8e5c-2e2bedf3ffcb admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:45.454445 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb0bfbe-5618-49c4-8e5c-2e2bedf3ffcb admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:45.455096 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb0bfbe-5618-49c4-8e5c-2e2bedf3ffcb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:45.456891 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb0bfbe-5618-49c4-8e5c-2e2bedf3ffcb admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:45.469718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1328/2671] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:23:45 2022] GET /identity/v3/roles => generated 1427 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 16:23:45.474438 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ce6200-b5bd-48d1-81b1-251583dd30dc admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:45.486379 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ce6200-b5bd-48d1-81b1-251583dd30dc admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:45.492434 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7dfd53-1012-428c-85f6-e4db7c049539 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:45.493051 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7dfd53-1012-428c-85f6-e4db7c049539 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:45.493331 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7dfd53-1012-428c-85f6-e4db7c049539 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:45.493606 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7dfd53-1012-428c-85f6-e4db7c049539 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:45.494729 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c7dfd53-1012-428c-85f6-e4db7c049539 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:45.496814 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c7dfd53-1012-428c-85f6-e4db7c049539 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:45.508930 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1344/2672] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:23:45 2022] GET /identity/v3/roles => generated 1427 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:23:45.514252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb0bfbe-5618-49c4-8e5c-2e2bedf3ffcb admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:45.526489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb0bfbe-5618-49c4-8e5c-2e2bedf3ffcb admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:45.533255 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abeec215-1f65-41eb-b70e-b81baa147baa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:45.533928 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abeec215-1f65-41eb-b70e-b81baa147baa admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:45.534211 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abeec215-1f65-41eb-b70e-b81baa147baa admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:45.534487 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abeec215-1f65-41eb-b70e-b81baa147baa admin admin] PATH_INFO: `/v3/projects/b277331c034b4813a414b75af757e04f/users/dc53d4d0861041eaa05de756f129929a/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:45.551267 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abeec215-1f65-41eb-b70e-b81baa147baa admin admin] RBAC: Authorizing `identity:create_grant(project_id=b277331c034b4813a414b75af757e04f, user_id=dc53d4d0861041eaa05de756f129929a, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:45.554026 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abeec215-1f65-41eb-b70e-b81baa147baa admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:45.579718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1329/2673] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:23:45 2022] PUT /identity/v3/projects/b277331c034b4813a414b75af757e04f/users/dc53d4d0861041eaa05de756f129929a/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:23:45.589294 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e67d11b5-5f46-4e57-9101-dd365fe8c5d5 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:45.589816 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e67d11b5-5f46-4e57-9101-dd365fe8c5d5 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:45.590917 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e67d11b5-5f46-4e57-9101-dd365fe8c5d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:45.632062 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-e67d11b5-5f46-4e57-9101-dd365fe8c5d5 None None] MFA Rules not processed for user `dc53d4d0861041eaa05de756f129929a`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:45.634302 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-e67d11b5-5f46-4e57-9101-dd365fe8c5d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:45.881546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1345/2674] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:45 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 294 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 16:23:45.887495 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abeec215-1f65-41eb-b70e-b81baa147baa admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:45.969063 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abeec215-1f65-41eb-b70e-b81baa147baa admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:45.975073 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dffc7da-3e97-493b-9247-5300bb691eb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:45.976019 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dffc7da-3e97-493b-9247-5300bb691eb9 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:45.976307 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dffc7da-3e97-493b-9247-5300bb691eb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:45.976587 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dffc7da-3e97-493b-9247-5300bb691eb9 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:45.977712 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dffc7da-3e97-493b-9247-5300bb691eb9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:45.981847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dffc7da-3e97-493b-9247-5300bb691eb9 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:45.998258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e67d11b5-5f46-4e57-9101-dd365fe8c5d5 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:46.000610 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-e67d11b5-5f46-4e57-9101-dd365fe8c5d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:46.022727 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1330/2675] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:23:45 2022] POST /identity/v3/projects => generated 392 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:23:46.026932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dffc7da-3e97-493b-9247-5300bb691eb9 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:46.028923 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:46.148182 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dffc7da-3e97-493b-9247-5300bb691eb9 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:46.151948 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e67d11b5-5f46-4e57-9101-dd365fe8c5d5 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:46.158053 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5c5da0-4435-483b-9ed5-86d3bfd36961 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:46.158683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5c5da0-4435-483b-9ed5-86d3bfd36961 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:46.159616 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5c5da0-4435-483b-9ed5-86d3bfd36961 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:46.159616 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5c5da0-4435-483b-9ed5-86d3bfd36961 admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:46.160012 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5c5da0-4435-483b-9ed5-86d3bfd36961 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:46.162167 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5c5da0-4435-483b-9ed5-86d3bfd36961 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:46.189780 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1331/2676] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:23:46 2022] POST /identity/v3/users => generated 303 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:23:46.196255 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5c5da0-4435-483b-9ed5-86d3bfd36961 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:46.216503 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5c5da0-4435-483b-9ed5-86d3bfd36961 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:46.223757 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a532ee2f-0829-40ab-9571-e26fb341c582 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:46.224500 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a532ee2f-0829-40ab-9571-e26fb341c582 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:46.224644 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a532ee2f-0829-40ab-9571-e26fb341c582 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:46.224931 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a532ee2f-0829-40ab-9571-e26fb341c582 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:46.225730 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a532ee2f-0829-40ab-9571-e26fb341c582 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:46.227443 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a532ee2f-0829-40ab-9571-e26fb341c582 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:46.236291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:46.240357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5c8684-fc2f-4815-ad71-04fa4b9faeff service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:46.241239 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5c8684-fc2f-4815-ad71-04fa4b9faeff service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:46.241239 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5c8684-fc2f-4815-ad71-04fa4b9faeff service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:46.241523 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5c8684-fc2f-4815-ad71-04fa4b9faeff service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:46.243380 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-0c5c8684-fc2f-4815-ad71-04fa4b9faeff service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:46.254404 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1332/2677] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:23:46 2022] GET /identity/v3/roles => generated 1427 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 16:23:46.255558 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a532ee2f-0829-40ab-9571-e26fb341c582 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:46.257321 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5c8684-fc2f-4815-ad71-04fa4b9faeff service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:46.259009 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5c8684-fc2f-4815-ad71-04fa4b9faeff service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:46.266658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a532ee2f-0829-40ab-9571-e26fb341c582 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:46.271620 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fbf8e3-d5e5-4725-848f-736307bf49d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:46.272324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fbf8e3-d5e5-4725-848f-736307bf49d3 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:46.272737 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fbf8e3-d5e5-4725-848f-736307bf49d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:46.272846 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fbf8e3-d5e5-4725-848f-736307bf49d3 admin admin] PATH_INFO: `/v3/projects/2393c70881be4d33897d9b7c05d366da/users/1503444cabde496c9eca370fea1ff6d6/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:46.275763 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1346/2678] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:23:45 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 276 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:23:46.291606 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04fbf8e3-d5e5-4725-848f-736307bf49d3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2393c70881be4d33897d9b7c05d366da, user_id=1503444cabde496c9eca370fea1ff6d6, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:46.297410 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04fbf8e3-d5e5-4725-848f-736307bf49d3 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:46.317884 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1333/2679] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:23:46 2022] PUT /identity/v3/projects/2393c70881be4d33897d9b7c05d366da/users/1503444cabde496c9eca370fea1ff6d6/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:46.322822 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5c8684-fc2f-4815-ad71-04fa4b9faeff service nova] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:46.333423 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5c8684-fc2f-4815-ad71-04fa4b9faeff service nova] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:46.395668 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8695850-e2fc-4d48-858d-a55b65657bb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:46.396423 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8695850-e2fc-4d48-858d-a55b65657bb4 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:46.396705 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8695850-e2fc-4d48-858d-a55b65657bb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:46.396973 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8695850-e2fc-4d48-858d-a55b65657bb4 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:46.397616 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8695850-e2fc-4d48-858d-a55b65657bb4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:46.399252 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8695850-e2fc-4d48-858d-a55b65657bb4 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:46.409804 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1347/2680] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:23:46 2022] GET /identity/v3/roles => generated 1427 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:23:46.413888 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fbf8e3-d5e5-4725-848f-736307bf49d3 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:46.438317 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fbf8e3-d5e5-4725-848f-736307bf49d3 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:46.446496 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed08b464-0aae-425f-b868-b2eeb4f1e436 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:46.447440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed08b464-0aae-425f-b868-b2eeb4f1e436 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:46.448517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed08b464-0aae-425f-b868-b2eeb4f1e436 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:46.449050 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed08b464-0aae-425f-b868-b2eeb4f1e436 admin admin] PATH_INFO: `/v3/projects/2393c70881be4d33897d9b7c05d366da/users/1503444cabde496c9eca370fea1ff6d6/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:46.452543 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed08b464-0aae-425f-b868-b2eeb4f1e436 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2393c70881be4d33897d9b7c05d366da, user_id=1503444cabde496c9eca370fea1ff6d6, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:46.457678 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed08b464-0aae-425f-b868-b2eeb4f1e436 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:46.476388 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1334/2681] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:23:46 2022] PUT /identity/v3/projects/2393c70881be4d33897d9b7c05d366da/users/1503444cabde496c9eca370fea1ff6d6/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:46.493346 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f40ce26-90cb-44b8-a629-48da85923db7 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:46.493844 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f40ce26-90cb-44b8-a629-48da85923db7 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:46.494200 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f40ce26-90cb-44b8-a629-48da85923db7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:46.523904 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-1f40ce26-90cb-44b8-a629-48da85923db7 None None] MFA Rules not processed for user `1503444cabde496c9eca370fea1ff6d6`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:46.525269 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-1f40ce26-90cb-44b8-a629-48da85923db7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:46.552670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=100 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:46.613229 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1348/2682] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:46 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:23:46.626664 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed08b464-0aae-425f-b868-b2eeb4f1e436 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:46.628633 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-ed08b464-0aae-425f-b868-b2eeb4f1e436 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:46.694747 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed08b464-0aae-425f-b868-b2eeb4f1e436 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:46.700113 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0347ab64-ac28-433a-bff2-39ff411305ac service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:46.700735 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0347ab64-ac28-433a-bff2-39ff411305ac service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:46.700997 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0347ab64-ac28-433a-bff2-39ff411305ac service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:46.701262 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0347ab64-ac28-433a-bff2-39ff411305ac service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:46.712143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0347ab64-ac28-433a-bff2-39ff411305ac service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:46.713585 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0347ab64-ac28-433a-bff2-39ff411305ac service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:46.724980 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1335/2683] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:23:46 2022] GET /identity/v3/auth/tokens => generated 3345 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:23:46.782418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f40ce26-90cb-44b8-a629-48da85923db7 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:46.794014 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f40ce26-90cb-44b8-a629-48da85923db7 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:46.797942 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0d8be0-1820-4b23-be56-210036046bfc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:46.798556 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0d8be0-1820-4b23-be56-210036046bfc service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:46.798837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0d8be0-1820-4b23-be56-210036046bfc service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:46.799107 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0d8be0-1820-4b23-be56-210036046bfc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:46.907007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0d8be0-1820-4b23-be56-210036046bfc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:46.908253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0d8be0-1820-4b23-be56-210036046bfc service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:46.923734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1349/2684] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:23:46 2022] GET /identity/v3/auth/tokens => generated 3280 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:23:49.321922 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0347ab64-ac28-433a-bff2-39ff411305ac service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:49.323527 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-0347ab64-ac28-433a-bff2-39ff411305ac service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:49.389663 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0347ab64-ac28-433a-bff2-39ff411305ac service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:49.394196 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c9c06f-2935-44e8-a59e-b34626be7244 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:49.394809 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c9c06f-2935-44e8-a59e-b34626be7244 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:49.395084 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c9c06f-2935-44e8-a59e-b34626be7244 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:49.395348 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c9c06f-2935-44e8-a59e-b34626be7244 admin admin] PATH_INFO: `/v3/users/c0d32d5a4e4045cf97019a66a7d0c0d3` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:49.396479 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c9c06f-2935-44e8-a59e-b34626be7244 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c0d32d5a4e4045cf97019a66a7d0c0d3)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:49.398368 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c9c06f-2935-44e8-a59e-b34626be7244 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:49.445040 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d4c9c06f-2935-44e8-a59e-b34626be7244 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c0d32d5a4e4045cf97019a66a7d0c0d3'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:49.445734 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d4c9c06f-2935-44e8-a59e-b34626be7244 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c0d32d5a4e4045cf97019a66a7d0c0d3'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:49.452646 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d4c9c06f-2935-44e8-a59e-b34626be7244 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c0d32d5a4e4045cf97019a66a7d0c0d3'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:49.460184 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=90,keystone:INSERT=2,keystone:DELETE=8 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:49.465449 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d4c9c06f-2935-44e8-a59e-b34626be7244 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c0d32d5a4e4045cf97019a66a7d0c0d3'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:49.466763 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d4c9c06f-2935-44e8-a59e-b34626be7244 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c0d32d5a4e4045cf97019a66a7d0c0d3'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:49.480658 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1336/2685] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:23:49 2022] DELETE /identity/v3/users/c0d32d5a4e4045cf97019a66a7d0c0d3 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:49.485849 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe80b4b-22b8-4ab8-a4f4-82b8cb127377 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:49.486339 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe80b4b-22b8-4ab8-a4f4-82b8cb127377 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:49.486444 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe80b4b-22b8-4ab8-a4f4-82b8cb127377 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:49.504681 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-cfe80b4b-22b8-4ab8-a4f4-82b8cb127377 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:49.505894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-cfe80b4b-22b8-4ab8-a4f4-82b8cb127377 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:49.564483 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1350/2686] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:49 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:23:49.574191 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c9c06f-2935-44e8-a59e-b34626be7244 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:49.575567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-d4c9c06f-2935-44e8-a59e-b34626be7244 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:49.634581 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c9c06f-2935-44e8-a59e-b34626be7244 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:49.639000 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caac47a7-4255-476a-ae3b-4020e07c1513 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:49.639598 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caac47a7-4255-476a-ae3b-4020e07c1513 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:49.639907 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caac47a7-4255-476a-ae3b-4020e07c1513 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:49.640173 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caac47a7-4255-476a-ae3b-4020e07c1513 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:49.649146 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caac47a7-4255-476a-ae3b-4020e07c1513 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:49.650643 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caac47a7-4255-476a-ae3b-4020e07c1513 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:49.661735 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1337/2687] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:23:49 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:23:49.946818 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe80b4b-22b8-4ab8-a4f4-82b8cb127377 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:49.948551 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-cfe80b4b-22b8-4ab8-a4f4-82b8cb127377 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:49.961317 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe80b4b-22b8-4ab8-a4f4-82b8cb127377 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:49.966633 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c1372e-6140-44ae-b2f0-bda99865d6b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:49.967267 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c1372e-6140-44ae-b2f0-bda99865d6b0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:49.967544 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c1372e-6140-44ae-b2f0-bda99865d6b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:49.967852 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c1372e-6140-44ae-b2f0-bda99865d6b0 admin admin] PATH_INFO: `/v3/projects/e3593174124f432fbcc8e026c014ca10` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:49.969015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c1372e-6140-44ae-b2f0-bda99865d6b0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e3593174124f432fbcc8e026c014ca10)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:49.971851 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c1372e-6140-44ae-b2f0-bda99865d6b0 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:50.021640 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-34c1372e-6140-44ae-b2f0-bda99865d6b0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e3593174124f432fbcc8e026c014ca10', 'request_id': 'req-34c1372e-6140-44ae-b2f0-bda99865d6b0'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:50.023460 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:50.027980 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-34c1372e-6140-44ae-b2f0-bda99865d6b0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e3593174124f432fbcc8e026c014ca10', 'request_id': 'req-34c1372e-6140-44ae-b2f0-bda99865d6b0'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:50.035345 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-34c1372e-6140-44ae-b2f0-bda99865d6b0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e3593174124f432fbcc8e026c014ca10', 'request_id': 'req-34c1372e-6140-44ae-b2f0-bda99865d6b0'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:50.049064 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-34c1372e-6140-44ae-b2f0-bda99865d6b0 admin admin] The token cache is being invalidate because project e3593174124f432fbcc8e026c014ca10 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:23:50.049356 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-34c1372e-6140-44ae-b2f0-bda99865d6b0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:50.049993 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-34c1372e-6140-44ae-b2f0-bda99865d6b0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:50.051483 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1351/2688] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:23:49 2022] DELETE /identity/v3/projects/e3593174124f432fbcc8e026c014ca10 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:50.055702 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caac47a7-4255-476a-ae3b-4020e07c1513 service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:50.065953 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caac47a7-4255-476a-ae3b-4020e07c1513 service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:50.123984 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70434b56-5b11-4b8d-a6f0-5ad0b3a14a36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:50.124694 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70434b56-5b11-4b8d-a6f0-5ad0b3a14a36 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:50.125045 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70434b56-5b11-4b8d-a6f0-5ad0b3a14a36 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:50.125389 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70434b56-5b11-4b8d-a6f0-5ad0b3a14a36 admin admin] PATH_INFO: `/v3/users/8cd8822f60a749a2972c54fb10a102a3` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:50.126632 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70434b56-5b11-4b8d-a6f0-5ad0b3a14a36 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8cd8822f60a749a2972c54fb10a102a3)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:50.128662 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70434b56-5b11-4b8d-a6f0-5ad0b3a14a36 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:50.178008 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-70434b56-5b11-4b8d-a6f0-5ad0b3a14a36 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8cd8822f60a749a2972c54fb10a102a3'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:50.178663 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-70434b56-5b11-4b8d-a6f0-5ad0b3a14a36 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8cd8822f60a749a2972c54fb10a102a3'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:50.186283 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-70434b56-5b11-4b8d-a6f0-5ad0b3a14a36 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8cd8822f60a749a2972c54fb10a102a3'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:50.200358 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-70434b56-5b11-4b8d-a6f0-5ad0b3a14a36 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8cd8822f60a749a2972c54fb10a102a3'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:50.201494 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-70434b56-5b11-4b8d-a6f0-5ad0b3a14a36 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8cd8822f60a749a2972c54fb10a102a3'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:50.215076 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1338/2689] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:23:50 2022] DELETE /identity/v3/users/8cd8822f60a749a2972c54fb10a102a3 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:23:50.274272 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c1372e-6140-44ae-b2f0-bda99865d6b0 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:50.275821 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-34c1372e-6140-44ae-b2f0-bda99865d6b0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:50.290286 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c1372e-6140-44ae-b2f0-bda99865d6b0 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:50.357338 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b316b1ad-dd45-4abf-8fcd-020fe0d6086a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:50.358136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b316b1ad-dd45-4abf-8fcd-020fe0d6086a admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:50.358524 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b316b1ad-dd45-4abf-8fcd-020fe0d6086a admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:50.358879 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b316b1ad-dd45-4abf-8fcd-020fe0d6086a admin admin] PATH_INFO: `/v3/users/dc53d4d0861041eaa05de756f129929a` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:50.360236 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b316b1ad-dd45-4abf-8fcd-020fe0d6086a admin admin] RBAC: Authorizing `identity:delete_user(user_id=dc53d4d0861041eaa05de756f129929a)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:50.362229 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b316b1ad-dd45-4abf-8fcd-020fe0d6086a admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:50.420587 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-b316b1ad-dd45-4abf-8fcd-020fe0d6086a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dc53d4d0861041eaa05de756f129929a'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:50.421388 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-b316b1ad-dd45-4abf-8fcd-020fe0d6086a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dc53d4d0861041eaa05de756f129929a'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:50.422102 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-b316b1ad-dd45-4abf-8fcd-020fe0d6086a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dc53d4d0861041eaa05de756f129929a'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:50.430162 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-b316b1ad-dd45-4abf-8fcd-020fe0d6086a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dc53d4d0861041eaa05de756f129929a'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:50.443735 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-b316b1ad-dd45-4abf-8fcd-020fe0d6086a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dc53d4d0861041eaa05de756f129929a'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:50.457455 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1352/2690] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:23:50 2022] DELETE /identity/v3/users/dc53d4d0861041eaa05de756f129929a => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:50.462081 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fea45d3-df6b-4acb-93c1-2d853ce2cd08 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:50.462567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fea45d3-df6b-4acb-93c1-2d853ce2cd08 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:50.462666 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fea45d3-df6b-4acb-93c1-2d853ce2cd08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:50.478598 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:50.485195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-5fea45d3-df6b-4acb-93c1-2d853ce2cd08 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:50.487217 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-5fea45d3-df6b-4acb-93c1-2d853ce2cd08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:50.552549 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1339/2691] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:50 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 16:23:50.564646 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b316b1ad-dd45-4abf-8fcd-020fe0d6086a admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:50.566092 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-b316b1ad-dd45-4abf-8fcd-020fe0d6086a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:50.620426 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fea45d3-df6b-4acb-93c1-2d853ce2cd08 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:50.623304 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-5fea45d3-df6b-4acb-93c1-2d853ce2cd08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:50.634487 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b316b1ad-dd45-4abf-8fcd-020fe0d6086a admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:50.636077 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fea45d3-df6b-4acb-93c1-2d853ce2cd08 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:50.644518 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9406df49-d4d1-423f-a008-8f244907efcc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:50.645262 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9406df49-d4d1-423f-a008-8f244907efcc service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:50.645610 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9406df49-d4d1-423f-a008-8f244907efcc service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:50.645969 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9406df49-d4d1-423f-a008-8f244907efcc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:50.646626 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af27d7ac-c924-4753-90cf-ba68880884de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:50.647293 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af27d7ac-c924-4753-90cf-ba68880884de admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:50.647576 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af27d7ac-c924-4753-90cf-ba68880884de admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:50.647878 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af27d7ac-c924-4753-90cf-ba68880884de admin admin] PATH_INFO: `/v3/projects/bfb363e22d92402ea36bc01526546eac` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:50.649228 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af27d7ac-c924-4753-90cf-ba68880884de admin admin] RBAC: Authorizing `identity:delete_project(project_id=bfb363e22d92402ea36bc01526546eac)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:50.651184 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af27d7ac-c924-4753-90cf-ba68880884de admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:50.657473 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9406df49-d4d1-423f-a008-8f244907efcc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:50.658989 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9406df49-d4d1-423f-a008-8f244907efcc service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:50.669129 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1353/2692] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:23:50 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:23:50.705833 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-af27d7ac-c924-4753-90cf-ba68880884de admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bfb363e22d92402ea36bc01526546eac', 'request_id': 'req-af27d7ac-c924-4753-90cf-ba68880884de'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:50.712858 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-af27d7ac-c924-4753-90cf-ba68880884de admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bfb363e22d92402ea36bc01526546eac', 'request_id': 'req-af27d7ac-c924-4753-90cf-ba68880884de'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:50.719930 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-af27d7ac-c924-4753-90cf-ba68880884de admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bfb363e22d92402ea36bc01526546eac', 'request_id': 'req-af27d7ac-c924-4753-90cf-ba68880884de'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:50.734799 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-af27d7ac-c924-4753-90cf-ba68880884de admin admin] The token cache is being invalidate because project bfb363e22d92402ea36bc01526546eac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:23:50.735091 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-af27d7ac-c924-4753-90cf-ba68880884de admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:50.735812 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-af27d7ac-c924-4753-90cf-ba68880884de admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:50.737332 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1340/2693] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:23:50 2022] DELETE /identity/v3/projects/bfb363e22d92402ea36bc01526546eac => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:50.741542 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9406df49-d4d1-423f-a008-8f244907efcc service neutron] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:50.747018 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:50.756317 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9406df49-d4d1-423f-a008-8f244907efcc service neutron] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:50.828641 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf895f2-f6e8-44b1-a51a-9d0eca247443 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:50.829292 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf895f2-f6e8-44b1-a51a-9d0eca247443 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:50.829576 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf895f2-f6e8-44b1-a51a-9d0eca247443 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:50.829855 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf895f2-f6e8-44b1-a51a-9d0eca247443 admin admin] PATH_INFO: `/v3/users/0e77c0e2e9be4ba4b383cd994a533dc9` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:50.831034 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf895f2-f6e8-44b1-a51a-9d0eca247443 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0e77c0e2e9be4ba4b383cd994a533dc9)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:50.832999 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf895f2-f6e8-44b1-a51a-9d0eca247443 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:50.889894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-ddf895f2-f6e8-44b1-a51a-9d0eca247443 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0e77c0e2e9be4ba4b383cd994a533dc9'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:50.891179 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-ddf895f2-f6e8-44b1-a51a-9d0eca247443 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0e77c0e2e9be4ba4b383cd994a533dc9'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:50.891897 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-ddf895f2-f6e8-44b1-a51a-9d0eca247443 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0e77c0e2e9be4ba4b383cd994a533dc9'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:50.900294 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-ddf895f2-f6e8-44b1-a51a-9d0eca247443 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0e77c0e2e9be4ba4b383cd994a533dc9'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:50.914380 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-ddf895f2-f6e8-44b1-a51a-9d0eca247443 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0e77c0e2e9be4ba4b383cd994a533dc9'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:50.930363 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1354/2694] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:23:50 2022] DELETE /identity/v3/users/0e77c0e2e9be4ba4b383cd994a533dc9 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:50.992101 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af27d7ac-c924-4753-90cf-ba68880884de admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:50.994079 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-af27d7ac-c924-4753-90cf-ba68880884de admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:51.008650 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af27d7ac-c924-4753-90cf-ba68880884de admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:51.073547 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0193399-b61e-417a-85a8-e78086dafc24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:51.074055 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0193399-b61e-417a-85a8-e78086dafc24 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:51.074353 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0193399-b61e-417a-85a8-e78086dafc24 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:51.074631 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0193399-b61e-417a-85a8-e78086dafc24 admin admin] PATH_INFO: `/v3/projects/b277331c034b4813a414b75af757e04f` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:51.076013 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0193399-b61e-417a-85a8-e78086dafc24 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b277331c034b4813a414b75af757e04f)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:51.078123 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0193399-b61e-417a-85a8-e78086dafc24 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:51.120018 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:51.135727 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-e0193399-b61e-417a-85a8-e78086dafc24 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b277331c034b4813a414b75af757e04f', 'request_id': 'req-e0193399-b61e-417a-85a8-e78086dafc24'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:51.143184 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-e0193399-b61e-417a-85a8-e78086dafc24 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b277331c034b4813a414b75af757e04f', 'request_id': 'req-e0193399-b61e-417a-85a8-e78086dafc24'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:51.149450 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-e0193399-b61e-417a-85a8-e78086dafc24 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b277331c034b4813a414b75af757e04f', 'request_id': 'req-e0193399-b61e-417a-85a8-e78086dafc24'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:51.164266 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-e0193399-b61e-417a-85a8-e78086dafc24 admin admin] The token cache is being invalidate because project b277331c034b4813a414b75af757e04f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:23:51.164563 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-e0193399-b61e-417a-85a8-e78086dafc24 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:51.165245 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-e0193399-b61e-417a-85a8-e78086dafc24 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:51.166963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1341/2695] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:23:50 2022] DELETE /identity/v3/projects/b277331c034b4813a414b75af757e04f => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:51.171483 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf895f2-f6e8-44b1-a51a-9d0eca247443 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:51.173045 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-ddf895f2-f6e8-44b1-a51a-9d0eca247443 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:51.186887 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf895f2-f6e8-44b1-a51a-9d0eca247443 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:51.244486 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be69b11-ff6b-4ed9-902f-9a6104e8c7c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:51.245133 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be69b11-ff6b-4ed9-902f-9a6104e8c7c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:51.245420 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be69b11-ff6b-4ed9-902f-9a6104e8c7c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:51.245729 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be69b11-ff6b-4ed9-902f-9a6104e8c7c9 admin admin] PATH_INFO: `/v3/users/1503444cabde496c9eca370fea1ff6d6` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:51.246888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be69b11-ff6b-4ed9-902f-9a6104e8c7c9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1503444cabde496c9eca370fea1ff6d6)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:51.248671 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0193399-b61e-417a-85a8-e78086dafc24 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:51.250007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be69b11-ff6b-4ed9-902f-9a6104e8c7c9 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:51.250308 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-e0193399-b61e-417a-85a8-e78086dafc24 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:51.266181 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0193399-b61e-417a-85a8-e78086dafc24 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:51.271115 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a4a349-0d9e-491b-8753-2a89e50ac02a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:51.271841 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a4a349-0d9e-491b-8753-2a89e50ac02a admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:51.272195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a4a349-0d9e-491b-8753-2a89e50ac02a admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:51.273158 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a4a349-0d9e-491b-8753-2a89e50ac02a admin admin] PATH_INFO: `/v3/projects/e06adf1f4c764d8897db902995eab4ba` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:51.274169 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a4a349-0d9e-491b-8753-2a89e50ac02a admin admin] RBAC: Authorizing `identity:delete_project(project_id=e06adf1f4c764d8897db902995eab4ba)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:51.276351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a4a349-0d9e-491b-8753-2a89e50ac02a admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:51.280578 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:51.311303 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0be69b11-ff6b-4ed9-902f-9a6104e8c7c9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1503444cabde496c9eca370fea1ff6d6'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:51.312058 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0be69b11-ff6b-4ed9-902f-9a6104e8c7c9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1503444cabde496c9eca370fea1ff6d6'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:51.312665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0be69b11-ff6b-4ed9-902f-9a6104e8c7c9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1503444cabde496c9eca370fea1ff6d6'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:51.320359 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0be69b11-ff6b-4ed9-902f-9a6104e8c7c9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1503444cabde496c9eca370fea1ff6d6'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:51.325972 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-31a4a349-0d9e-491b-8753-2a89e50ac02a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e06adf1f4c764d8897db902995eab4ba', 'request_id': 'req-31a4a349-0d9e-491b-8753-2a89e50ac02a'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:51.332266 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-31a4a349-0d9e-491b-8753-2a89e50ac02a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e06adf1f4c764d8897db902995eab4ba', 'request_id': 'req-31a4a349-0d9e-491b-8753-2a89e50ac02a'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:51.332941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0be69b11-ff6b-4ed9-902f-9a6104e8c7c9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1503444cabde496c9eca370fea1ff6d6'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:51.337752 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-31a4a349-0d9e-491b-8753-2a89e50ac02a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e06adf1f4c764d8897db902995eab4ba', 'request_id': 'req-31a4a349-0d9e-491b-8753-2a89e50ac02a'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:51.346190 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1355/2696] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:23:51 2022] DELETE /identity/v3/users/1503444cabde496c9eca370fea1ff6d6 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:51.354141 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-31a4a349-0d9e-491b-8753-2a89e50ac02a admin admin] The token cache is being invalidate because project e06adf1f4c764d8897db902995eab4ba was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:23:51.354504 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-31a4a349-0d9e-491b-8753-2a89e50ac02a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:51.355426 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-31a4a349-0d9e-491b-8753-2a89e50ac02a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:51.358520 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1342/2697] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:23:51 2022] DELETE /identity/v3/projects/e06adf1f4c764d8897db902995eab4ba => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:23:51.363344 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff38078d-7908-40c1-89fa-a92c6a60bd8b None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:51.363698 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff38078d-7908-40c1-89fa-a92c6a60bd8b None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:51.364151 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff38078d-7908-40c1-89fa-a92c6a60bd8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:51.384533 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-ff38078d-7908-40c1-89fa-a92c6a60bd8b None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:51.385840 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-ff38078d-7908-40c1-89fa-a92c6a60bd8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:51.448426 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1356/2698] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:51 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:23:51.457077 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df20c86-8d6e-4da2-888e-81670cbe8409 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:51.457406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df20c86-8d6e-4da2-888e-81670cbe8409 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:51.457727 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df20c86-8d6e-4da2-888e-81670cbe8409 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:51.483195 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-2df20c86-8d6e-4da2-888e-81670cbe8409 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:51.484804 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-2df20c86-8d6e-4da2-888e-81670cbe8409 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:51.494225 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1343/2699] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:51 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:23:51.498883 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff38078d-7908-40c1-89fa-a92c6a60bd8b None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:51.516159 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff38078d-7908-40c1-89fa-a92c6a60bd8b None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:51.520893 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98a96ba-7e31-4fc9-9005-eafcc837a525 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:51.521644 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98a96ba-7e31-4fc9-9005-eafcc837a525 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:51.521932 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98a96ba-7e31-4fc9-9005-eafcc837a525 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:51.522202 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98a96ba-7e31-4fc9-9005-eafcc837a525 admin admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:51.522858 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e98a96ba-7e31-4fc9-9005-eafcc837a525 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 16:23:51.523165 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e98a96ba-7e31-4fc9-9005-eafcc837a525 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:51.524873 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e98a96ba-7e31-4fc9-9005-eafcc837a525 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:51.538640 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1357/2700] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 16:23:51 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 16:23:51.543082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df20c86-8d6e-4da2-888e-81670cbe8409 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:51.553427 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df20c86-8d6e-4da2-888e-81670cbe8409 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:51.558210 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e40f2c-14cc-4eef-85ba-2d2368ccc23b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:51.558807 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e40f2c-14cc-4eef-85ba-2d2368ccc23b admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:51.559088 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e40f2c-14cc-4eef-85ba-2d2368ccc23b admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:51.559365 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e40f2c-14cc-4eef-85ba-2d2368ccc23b admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:51.560122 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e40f2c-14cc-4eef-85ba-2d2368ccc23b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:51.562044 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e40f2c-14cc-4eef-85ba-2d2368ccc23b admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:51.593391 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1344/2701] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:23:51 2022] POST /identity/v3/projects => generated 374 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:23:51.597292 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98a96ba-7e31-4fc9-9005-eafcc837a525 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:51.609233 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98a96ba-7e31-4fc9-9005-eafcc837a525 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:51.646374 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e40f2c-14cc-4eef-85ba-2d2368ccc23b admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:51.648960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-f1e40f2c-14cc-4eef-85ba-2d2368ccc23b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:51.688065 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c9c9389-bc7f-4023-8da8-88c52e0a5b84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:51.688511 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9c9389-bc7f-4023-8da8-88c52e0a5b84 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:51.688882 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9c9389-bc7f-4023-8da8-88c52e0a5b84 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:51.689227 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9c9389-bc7f-4023-8da8-88c52e0a5b84 admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:51.689972 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c9c9389-bc7f-4023-8da8-88c52e0a5b84 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:51.691968 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c9c9389-bc7f-4023-8da8-88c52e0a5b84 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:51.716486 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1358/2702] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:23:51 2022] POST /identity/v3/users => generated 288 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 16:23:51.720144 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c9c9389-bc7f-4023-8da8-88c52e0a5b84 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:51.730018 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c9c9389-bc7f-4023-8da8-88c52e0a5b84 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:51.735352 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ffc7946-598b-47ff-98cd-836add7d3b06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:51.736036 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffc7946-598b-47ff-98cd-836add7d3b06 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:51.736320 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffc7946-598b-47ff-98cd-836add7d3b06 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:51.736630 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffc7946-598b-47ff-98cd-836add7d3b06 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:51.737253 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ffc7946-598b-47ff-98cd-836add7d3b06 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:51.738853 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ffc7946-598b-47ff-98cd-836add7d3b06 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:51.749350 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e40f2c-14cc-4eef-85ba-2d2368ccc23b admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:51.753761 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1359/2703] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:23:51 2022] GET /identity/v3/roles => generated 1427 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 16:23:51.754684 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f909b6fb-ed97-4976-9d2d-f11ee2c3c823 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:51.755014 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ffc7946-598b-47ff-98cd-836add7d3b06 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:51.755479 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f909b6fb-ed97-4976-9d2d-f11ee2c3c823 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:51.755880 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f909b6fb-ed97-4976-9d2d-f11ee2c3c823 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:51.756471 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f909b6fb-ed97-4976-9d2d-f11ee2c3c823 admin admin] PATH_INFO: `/v3/projects/2393c70881be4d33897d9b7c05d366da` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:51.757873 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f909b6fb-ed97-4976-9d2d-f11ee2c3c823 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2393c70881be4d33897d9b7c05d366da)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:51.759982 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f909b6fb-ed97-4976-9d2d-f11ee2c3c823 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:51.766237 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ffc7946-598b-47ff-98cd-836add7d3b06 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:51.771159 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c3abfd-bae0-4c20-99f8-81059d1d7e3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:51.771802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c3abfd-bae0-4c20-99f8-81059d1d7e3e admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:51.772089 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c3abfd-bae0-4c20-99f8-81059d1d7e3e admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:51.772358 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c3abfd-bae0-4c20-99f8-81059d1d7e3e admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:51.773015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c3abfd-bae0-4c20-99f8-81059d1d7e3e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:51.774874 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c3abfd-bae0-4c20-99f8-81059d1d7e3e admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:51.776356 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=90,keystone:DELETE=6,keystone:INSERT=4 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:51.781474 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:51.800228 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1360/2704] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:23:51 2022] GET /identity/v3/roles => generated 1427 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:23:51.801606 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c3abfd-bae0-4c20-99f8-81059d1d7e3e admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:51.813172 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c3abfd-bae0-4c20-99f8-81059d1d7e3e admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:51.819844 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-f909b6fb-ed97-4976-9d2d-f11ee2c3c823 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2393c70881be4d33897d9b7c05d366da', 'request_id': 'req-f909b6fb-ed97-4976-9d2d-f11ee2c3c823'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:51.832417 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-f909b6fb-ed97-4976-9d2d-f11ee2c3c823 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2393c70881be4d33897d9b7c05d366da', 'request_id': 'req-f909b6fb-ed97-4976-9d2d-f11ee2c3c823'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:51.838693 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-f909b6fb-ed97-4976-9d2d-f11ee2c3c823 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2393c70881be4d33897d9b7c05d366da', 'request_id': 'req-f909b6fb-ed97-4976-9d2d-f11ee2c3c823'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:51.856676 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-f909b6fb-ed97-4976-9d2d-f11ee2c3c823 admin admin] The token cache is being invalidate because project 2393c70881be4d33897d9b7c05d366da was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:23:51.857132 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-f909b6fb-ed97-4976-9d2d-f11ee2c3c823 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:51.857946 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-f909b6fb-ed97-4976-9d2d-f11ee2c3c823 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:51.859685 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1345/2705] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:23:51 2022] DELETE /identity/v3/projects/2393c70881be4d33897d9b7c05d366da => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:23:51.864640 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a97fb4-392e-473d-a91b-8ed8510f9920 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:51.864928 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a97fb4-392e-473d-a91b-8ed8510f9920 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:51.865209 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a97fb4-392e-473d-a91b-8ed8510f9920 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:51.874138 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00df581-4c3a-4201-b9d1-f1b36b321308 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:51.874985 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00df581-4c3a-4201-b9d1-f1b36b321308 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:51.875272 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00df581-4c3a-4201-b9d1-f1b36b321308 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:51.875544 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00df581-4c3a-4201-b9d1-f1b36b321308 admin admin] PATH_INFO: `/v3/projects/4763981fec6b4e7a83e43f55b8078ef3/users/76a5787cf2f74294830403d4e68b9a44/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:51.886381 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-a2a97fb4-392e-473d-a91b-8ed8510f9920 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:51.887712 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-a2a97fb4-392e-473d-a91b-8ed8510f9920 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:51.891422 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a00df581-4c3a-4201-b9d1-f1b36b321308 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4763981fec6b4e7a83e43f55b8078ef3, user_id=76a5787cf2f74294830403d4e68b9a44, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:51.894134 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a00df581-4c3a-4201-b9d1-f1b36b321308 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:51.896188 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1346/2706] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:51 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 16:23:51.906193 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b734691-8bb8-49d0-822d-fc729b7e2c1d None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:51.906479 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b734691-8bb8-49d0-822d-fc729b7e2c1d None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:51.906776 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b734691-8bb8-49d0-822d-fc729b7e2c1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:51.907683 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1361/2707] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:23:51 2022] PUT /identity/v3/projects/4763981fec6b4e7a83e43f55b8078ef3/users/76a5787cf2f74294830403d4e68b9a44/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:51.918116 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380fad48-ce48-43e0-814a-ccc47f32a6fa None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:51.918116 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380fad48-ce48-43e0-814a-ccc47f32a6fa None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:51.918426 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380fad48-ce48-43e0-814a-ccc47f32a6fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:51.927071 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-6b734691-8bb8-49d0-822d-fc729b7e2c1d None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:51.928330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-6b734691-8bb8-49d0-822d-fc729b7e2c1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:51.947295 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-380fad48-ce48-43e0-814a-ccc47f32a6fa None None] MFA Rules not processed for user `76a5787cf2f74294830403d4e68b9a44`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:51.948533 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-380fad48-ce48-43e0-814a-ccc47f32a6fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:51.990105 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1347/2708] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:51 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 16:23:51.994208 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b734691-8bb8-49d0-822d-fc729b7e2c1d None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:52.004067 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b734691-8bb8-49d0-822d-fc729b7e2c1d None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:52.008775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ba06e2-866c-47ee-875f-429779c60fad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:52.009378 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ba06e2-866c-47ee-875f-429779c60fad admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:52.009655 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ba06e2-866c-47ee-875f-429779c60fad admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:52.009929 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ba06e2-866c-47ee-875f-429779c60fad admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:52.011202 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ba06e2-866c-47ee-875f-429779c60fad admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 16:23:52.011512 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ba06e2-866c-47ee-875f-429779c60fad admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:52.013229 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ba06e2-866c-47ee-875f-429779c60fad admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:52.016948 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1362/2709] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:51 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 16:23:52.029586 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1348/2710] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 16:23:51 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 16:23:52.029884 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380fad48-ce48-43e0-814a-ccc47f32a6fa None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:52.030642 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-380fad48-ce48-43e0-814a-ccc47f32a6fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:52.031149 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ba06e2-866c-47ee-875f-429779c60fad admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:52.042034 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ba06e2-866c-47ee-875f-429779c60fad admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:52.053517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a299f73-8702-4b89-9305-be995e27c238 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:52.054515 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a299f73-8702-4b89-9305-be995e27c238 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:52.054802 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a299f73-8702-4b89-9305-be995e27c238 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:52.055084 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a299f73-8702-4b89-9305-be995e27c238 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:52.056089 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a299f73-8702-4b89-9305-be995e27c238 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:52.058023 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a299f73-8702-4b89-9305-be995e27c238 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:52.088665 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1349/2711] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:23:52 2022] POST /identity/v3/projects => generated 384 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 16:23:52.093933 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a299f73-8702-4b89-9305-be995e27c238 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:52.094073 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380fad48-ce48-43e0-814a-ccc47f32a6fa None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:52.113857 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:52.143437 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bacdb930-b80b-4f38-b167-d1c833ff6054 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:52.144070 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacdb930-b80b-4f38-b167-d1c833ff6054 service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:52.144353 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacdb930-b80b-4f38-b167-d1c833ff6054 service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:52.144625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacdb930-b80b-4f38-b167-d1c833ff6054 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:52.160181 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a299f73-8702-4b89-9305-be995e27c238 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:52.164881 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89749423-e555-4a7a-8c64-cbfd954b2988 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:52.165484 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89749423-e555-4a7a-8c64-cbfd954b2988 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:52.165766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89749423-e555-4a7a-8c64-cbfd954b2988 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:52.166042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89749423-e555-4a7a-8c64-cbfd954b2988 admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:52.166673 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89749423-e555-4a7a-8c64-cbfd954b2988 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:52.168661 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89749423-e555-4a7a-8c64-cbfd954b2988 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:52.179302 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:52.195160 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1350/2712] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:23:52 2022] POST /identity/v3/users => generated 293 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 16:23:52.199121 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89749423-e555-4a7a-8c64-cbfd954b2988 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:52.205496 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bacdb930-b80b-4f38-b167-d1c833ff6054 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:52.206945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bacdb930-b80b-4f38-b167-d1c833ff6054 service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:52.209437 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89749423-e555-4a7a-8c64-cbfd954b2988 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:52.214422 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c089d08-000a-48ad-b7ac-47fb9b7a2b82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:52.215020 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c089d08-000a-48ad-b7ac-47fb9b7a2b82 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:52.215302 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c089d08-000a-48ad-b7ac-47fb9b7a2b82 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:52.215578 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c089d08-000a-48ad-b7ac-47fb9b7a2b82 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:52.216339 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1363/2713] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:23:52 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 189 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 16:23:52.216837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c089d08-000a-48ad-b7ac-47fb9b7a2b82 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:52.218448 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c089d08-000a-48ad-b7ac-47fb9b7a2b82 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:52.233046 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1351/2714] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:23:52 2022] GET /identity/v3/roles => generated 1427 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 16:23:52.234340 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bacdb930-b80b-4f38-b167-d1c833ff6054 service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:52.241968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c089d08-000a-48ad-b7ac-47fb9b7a2b82 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:52.243719 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-6c089d08-000a-48ad-b7ac-47fb9b7a2b82 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:52.245534 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bacdb930-b80b-4f38-b167-d1c833ff6054 service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:52.251102 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b599d44-3f78-4576-818a-80636efb1143 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:52.251747 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b599d44-3f78-4576-818a-80636efb1143 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:52.252033 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b599d44-3f78-4576-818a-80636efb1143 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:52.252316 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b599d44-3f78-4576-818a-80636efb1143 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:52.252972 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b599d44-3f78-4576-818a-80636efb1143 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:52.254591 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b599d44-3f78-4576-818a-80636efb1143 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:52.257765 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c089d08-000a-48ad-b7ac-47fb9b7a2b82 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:52.262594 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c62a0eb-9f98-4ef6-9795-35e3c4a4b3b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:52.263979 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c62a0eb-9f98-4ef6-9795-35e3c4a4b3b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:52.264268 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c62a0eb-9f98-4ef6-9795-35e3c4a4b3b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:52.264546 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c62a0eb-9f98-4ef6-9795-35e3c4a4b3b1 admin admin] PATH_INFO: `/v3/users/76a5787cf2f74294830403d4e68b9a44` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:52.265714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c62a0eb-9f98-4ef6-9795-35e3c4a4b3b1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=76a5787cf2f74294830403d4e68b9a44)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:52.266683 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1364/2715] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:23:52 2022] GET /identity/v3/roles => generated 1427 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:23:52.267660 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c62a0eb-9f98-4ef6-9795-35e3c4a4b3b1 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:52.270673 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b599d44-3f78-4576-818a-80636efb1143 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:52.283099 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b599d44-3f78-4576-818a-80636efb1143 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:52.288770 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed77031-4dad-469e-8661-207dc7769ae4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:52.289519 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed77031-4dad-469e-8661-207dc7769ae4 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:52.289798 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed77031-4dad-469e-8661-207dc7769ae4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:52.290069 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed77031-4dad-469e-8661-207dc7769ae4 admin admin] PATH_INFO: `/v3/projects/7eb9a49bc8bf4fafa891c33ac3f7178d/users/6143ca6747fe481aa1cab63ed4897ed6/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:52.308652 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed77031-4dad-469e-8661-207dc7769ae4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7eb9a49bc8bf4fafa891c33ac3f7178d, user_id=6143ca6747fe481aa1cab63ed4897ed6, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:52.311682 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed77031-4dad-469e-8661-207dc7769ae4 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:52.323202 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-0c62a0eb-9f98-4ef6-9795-35e3c4a4b3b1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '76a5787cf2f74294830403d4e68b9a44'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:52.323911 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-0c62a0eb-9f98-4ef6-9795-35e3c4a4b3b1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '76a5787cf2f74294830403d4e68b9a44'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:52.325546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1365/2716] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:23:52 2022] PUT /identity/v3/projects/7eb9a49bc8bf4fafa891c33ac3f7178d/users/6143ca6747fe481aa1cab63ed4897ed6/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:52.331492 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-0c62a0eb-9f98-4ef6-9795-35e3c4a4b3b1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '76a5787cf2f74294830403d4e68b9a44'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:52.336351 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0629e0d1-a513-4509-9dfd-536da4477744 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:52.336637 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0629e0d1-a513-4509-9dfd-536da4477744 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:52.336916 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0629e0d1-a513-4509-9dfd-536da4477744 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:52.345232 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-0c62a0eb-9f98-4ef6-9795-35e3c4a4b3b1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '76a5787cf2f74294830403d4e68b9a44'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:52.345979 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-0c62a0eb-9f98-4ef6-9795-35e3c4a4b3b1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '76a5787cf2f74294830403d4e68b9a44'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:52.360606 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1352/2717] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:23:52 2022] DELETE /identity/v3/users/76a5787cf2f74294830403d4e68b9a44 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:23:52.365009 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1c47e1-827d-4197-805d-0031b0aa2ea1 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:52.365373 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1c47e1-827d-4197-805d-0031b0aa2ea1 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:52.365654 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1c47e1-827d-4197-805d-0031b0aa2ea1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:52.367991 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-0629e0d1-a513-4509-9dfd-536da4477744 None None] MFA Rules not processed for user `6143ca6747fe481aa1cab63ed4897ed6`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:52.369201 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-0629e0d1-a513-4509-9dfd-536da4477744 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:52.385668 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-dd1c47e1-827d-4197-805d-0031b0aa2ea1 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:52.386885 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-dd1c47e1-827d-4197-805d-0031b0aa2ea1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:52.437584 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1366/2718] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:52 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 16:23:52.442418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0629e0d1-a513-4509-9dfd-536da4477744 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:52.443878 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-0629e0d1-a513-4509-9dfd-536da4477744 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:52.449836 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1353/2719] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:52 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:23:52.460062 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1c47e1-827d-4197-805d-0031b0aa2ea1 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:52.460920 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0629e0d1-a513-4509-9dfd-536da4477744 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:52.461595 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-dd1c47e1-827d-4197-805d-0031b0aa2ea1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:52.468473 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc04d426-bd93-4e6a-89f2-72e73120a489 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:52.469179 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc04d426-bd93-4e6a-89f2-72e73120a489 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:52.469468 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc04d426-bd93-4e6a-89f2-72e73120a489 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:52.469818 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc04d426-bd93-4e6a-89f2-72e73120a489 admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:52.470472 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc04d426-bd93-4e6a-89f2-72e73120a489 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:52.472458 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc04d426-bd93-4e6a-89f2-72e73120a489 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:52.501518 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1367/2720] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:23:52 2022] POST /identity/v3/projects => generated 384 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:23:52.505526 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc04d426-bd93-4e6a-89f2-72e73120a489 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:52.518263 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:52.524931 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:INSERT=2,keystone:SELECT=88,keystone:DELETE=10 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:52.539126 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1c47e1-827d-4197-805d-0031b0aa2ea1 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:52.571692 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc04d426-bd93-4e6a-89f2-72e73120a489 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:52.577010 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2813320-500b-4467-a206-a81c6afb141d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:52.577624 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2813320-500b-4467-a206-a81c6afb141d admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:52.577938 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2813320-500b-4467-a206-a81c6afb141d admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:52.578219 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2813320-500b-4467-a206-a81c6afb141d admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:52.578849 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2813320-500b-4467-a206-a81c6afb141d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:52.580810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2813320-500b-4467-a206-a81c6afb141d admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:52.593047 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf6d54a-9a8d-4985-94c2-4b7396a35228 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:52.593668 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf6d54a-9a8d-4985-94c2-4b7396a35228 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:52.593986 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf6d54a-9a8d-4985-94c2-4b7396a35228 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:52.594265 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf6d54a-9a8d-4985-94c2-4b7396a35228 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:52.607561 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1368/2721] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:23:52 2022] POST /identity/v3/users => generated 299 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 16:23:52.608135 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf6d54a-9a8d-4985-94c2-4b7396a35228 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:52.608135 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf6d54a-9a8d-4985-94c2-4b7396a35228 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:52.609508 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2813320-500b-4467-a206-a81c6afb141d admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:52.616945 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1354/2722] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:23:52 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:23:52.619379 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2813320-500b-4467-a206-a81c6afb141d admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:52.626136 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc7e69c-42e1-4ccb-b6b8-adaf6104f6e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:52.626745 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc7e69c-42e1-4ccb-b6b8-adaf6104f6e4 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:52.627029 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc7e69c-42e1-4ccb-b6b8-adaf6104f6e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:52.627304 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc7e69c-42e1-4ccb-b6b8-adaf6104f6e4 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:52.627991 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc7e69c-42e1-4ccb-b6b8-adaf6104f6e4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:52.629595 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc7e69c-42e1-4ccb-b6b8-adaf6104f6e4 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:52.642841 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1369/2723] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:23:52 2022] GET /identity/v3/roles => generated 1427 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 16:23:52.647074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf6d54a-9a8d-4985-94c2-4b7396a35228 service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:52.657253 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf6d54a-9a8d-4985-94c2-4b7396a35228 service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:52.662412 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9cef70-02f8-4e83-9852-cf533a26e70c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:52.663063 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9cef70-02f8-4e83-9852-cf533a26e70c admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:52.663347 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9cef70-02f8-4e83-9852-cf533a26e70c admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:52.663618 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9cef70-02f8-4e83-9852-cf533a26e70c admin admin] PATH_INFO: `/v3/projects/61d01ca289c14b9987702695b5a41e73/users/c5c20a34220040a1b03428d2dbd3f988/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:52.682306 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9cef70-02f8-4e83-9852-cf533a26e70c admin admin] RBAC: Authorizing `identity:create_grant(project_id=61d01ca289c14b9987702695b5a41e73, user_id=c5c20a34220040a1b03428d2dbd3f988, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:52.686074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9cef70-02f8-4e83-9852-cf533a26e70c admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:52.699723 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1355/2724] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:23:52 2022] PUT /identity/v3/projects/61d01ca289c14b9987702695b5a41e73/users/c5c20a34220040a1b03428d2dbd3f988/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:52.703613 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc7e69c-42e1-4ccb-b6b8-adaf6104f6e4 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:52.713850 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc7e69c-42e1-4ccb-b6b8-adaf6104f6e4 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:52.790877 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e7694e-d556-429d-93ce-38c381aa4022 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:52.791502 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e7694e-d556-429d-93ce-38c381aa4022 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:52.791811 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e7694e-d556-429d-93ce-38c381aa4022 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:52.792088 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e7694e-d556-429d-93ce-38c381aa4022 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:52.792739 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e7694e-d556-429d-93ce-38c381aa4022 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:52.794479 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e7694e-d556-429d-93ce-38c381aa4022 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:52.804924 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1370/2725] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:23:52 2022] GET /identity/v3/roles => generated 1427 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:23:52.809224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9cef70-02f8-4e83-9852-cf533a26e70c admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:52.821026 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9cef70-02f8-4e83-9852-cf533a26e70c admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:52.826627 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09d75ba-86af-4016-aae4-2b77870fed15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:52.827406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09d75ba-86af-4016-aae4-2b77870fed15 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:52.827721 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09d75ba-86af-4016-aae4-2b77870fed15 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:52.828001 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09d75ba-86af-4016-aae4-2b77870fed15 admin admin] PATH_INFO: `/v3/projects/61d01ca289c14b9987702695b5a41e73/users/c5c20a34220040a1b03428d2dbd3f988/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:52.830529 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09d75ba-86af-4016-aae4-2b77870fed15 admin admin] RBAC: Authorizing `identity:create_grant(project_id=61d01ca289c14b9987702695b5a41e73, user_id=c5c20a34220040a1b03428d2dbd3f988, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:52.833290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09d75ba-86af-4016-aae4-2b77870fed15 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:52.847066 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1356/2726] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:23:52 2022] PUT /identity/v3/projects/61d01ca289c14b9987702695b5a41e73/users/c5c20a34220040a1b03428d2dbd3f988/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:52.858844 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c11d28c-30a8-4f47-8735-a980765f724e None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:52.858844 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c11d28c-30a8-4f47-8735-a980765f724e None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:52.859270 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c11d28c-30a8-4f47-8735-a980765f724e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:52.890215 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-7c11d28c-30a8-4f47-8735-a980765f724e None None] MFA Rules not processed for user `c5c20a34220040a1b03428d2dbd3f988`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:23:52.891480 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-7c11d28c-30a8-4f47-8735-a980765f724e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:52.973556 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1371/2727] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:23:52 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 16:23:52.985265 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09d75ba-86af-4016-aae4-2b77870fed15 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:52.985511 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c11d28c-30a8-4f47-8735-a980765f724e None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:52.987127 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-7c11d28c-30a8-4f47-8735-a980765f724e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:52.987772 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-d09d75ba-86af-4016-aae4-2b77870fed15 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:53.002519 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:53.072632 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c11d28c-30a8-4f47-8735-a980765f724e None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:53.074342 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09d75ba-86af-4016-aae4-2b77870fed15 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:53.078441 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07117d22-77cb-4943-9496-bb0f7765961b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:53.079050 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07117d22-77cb-4943-9496-bb0f7765961b service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:53.079330 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07117d22-77cb-4943-9496-bb0f7765961b service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:53.079602 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07117d22-77cb-4943-9496-bb0f7765961b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:53.080054 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8d0534-33f9-418c-8943-1a91c787c98c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:53.080908 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8d0534-33f9-418c-8943-1a91c787c98c admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:53.081290 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8d0534-33f9-418c-8943-1a91c787c98c admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:53.081735 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8d0534-33f9-418c-8943-1a91c787c98c admin admin] PATH_INFO: `/v3/projects/4763981fec6b4e7a83e43f55b8078ef3` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:53.083147 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e8d0534-33f9-418c-8943-1a91c787c98c admin admin] RBAC: Authorizing `identity:delete_project(project_id=4763981fec6b4e7a83e43f55b8078ef3)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:53.086321 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e8d0534-33f9-418c-8943-1a91c787c98c admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:53.142894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-2e8d0534-33f9-418c-8943-1a91c787c98c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4763981fec6b4e7a83e43f55b8078ef3', 'request_id': 'req-2e8d0534-33f9-418c-8943-1a91c787c98c'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:53.144807 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07117d22-77cb-4943-9496-bb0f7765961b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:53.146247 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07117d22-77cb-4943-9496-bb0f7765961b service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:53.149328 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-2e8d0534-33f9-418c-8943-1a91c787c98c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4763981fec6b4e7a83e43f55b8078ef3', 'request_id': 'req-2e8d0534-33f9-418c-8943-1a91c787c98c'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:53.155279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-2e8d0534-33f9-418c-8943-1a91c787c98c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4763981fec6b4e7a83e43f55b8078ef3', 'request_id': 'req-2e8d0534-33f9-418c-8943-1a91c787c98c'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:53.155551 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1357/2728] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:23:52 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 171 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:23:53.168837 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-2e8d0534-33f9-418c-8943-1a91c787c98c admin admin] The token cache is being invalidate because project 4763981fec6b4e7a83e43f55b8078ef3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:23:53.169168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-2e8d0534-33f9-418c-8943-1a91c787c98c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:53.169884 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-2e8d0534-33f9-418c-8943-1a91c787c98c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:23:53.171512 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1372/2729] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:23:52 2022] DELETE /identity/v3/projects/4763981fec6b4e7a83e43f55b8078ef3 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:23:57.423839 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07117d22-77cb-4943-9496-bb0f7765961b service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:57.425278 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-07117d22-77cb-4943-9496-bb0f7765961b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:57.436904 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07117d22-77cb-4943-9496-bb0f7765961b service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:57.453249 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:23:57.490321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e659afa4-43bd-4a68-82d2-2825e4ae1cdf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:57.491132 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e659afa4-43bd-4a68-82d2-2825e4ae1cdf service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:57.491419 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e659afa4-43bd-4a68-82d2-2825e4ae1cdf service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:57.491731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e659afa4-43bd-4a68-82d2-2825e4ae1cdf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:57.493506 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-e659afa4-43bd-4a68-82d2-2825e4ae1cdf service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:57.563802 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e659afa4-43bd-4a68-82d2-2825e4ae1cdf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:57.565120 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e659afa4-43bd-4a68-82d2-2825e4ae1cdf service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:57.574906 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1358/2730] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:23:57 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:23:57.629636 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8d0534-33f9-418c-8943-1a91c787c98c admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:23:57.632330 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-2e8d0534-33f9-418c-8943-1a91c787c98c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:57.765444 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8d0534-33f9-418c-8943-1a91c787c98c admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:23:57.774701 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14bc2ac-410d-477e-8ab4-2d7df9e94a41 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:23:57.775606 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14bc2ac-410d-477e-8ab4-2d7df9e94a41 service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:23:57.776239 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14bc2ac-410d-477e-8ab4-2d7df9e94a41 service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:23:57.776780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14bc2ac-410d-477e-8ab4-2d7df9e94a41 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:23:57.779698 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-d14bc2ac-410d-477e-8ab4-2d7df9e94a41 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:23:57.799937 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14bc2ac-410d-477e-8ab4-2d7df9e94a41 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:23:57.802778 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14bc2ac-410d-477e-8ab4-2d7df9e94a41 service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:23:57.827417 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1373/2731] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:23:57 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 16:24:00.378745 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e659afa4-43bd-4a68-82d2-2825e4ae1cdf service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:00.380290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-e659afa4-43bd-4a68-82d2-2825e4ae1cdf service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:24:00.442873 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e659afa4-43bd-4a68-82d2-2825e4ae1cdf service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:00.446500 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7791f0b-4ab2-4ccd-8482-d965cc622456 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:00.447135 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7791f0b-4ab2-4ccd-8482-d965cc622456 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:00.447376 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7791f0b-4ab2-4ccd-8482-d965cc622456 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:00.447680 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7791f0b-4ab2-4ccd-8482-d965cc622456 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:00.449598 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-c7791f0b-4ab2-4ccd-8482-d965cc622456 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:24:00.519547 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7791f0b-4ab2-4ccd-8482-d965cc622456 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:00.522156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7791f0b-4ab2-4ccd-8482-d965cc622456 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:00.536837 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1359/2732] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:24:00 2022] GET /identity/v3/auth/tokens => generated 3333 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:24:02.430203 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14bc2ac-410d-477e-8ab4-2d7df9e94a41 service nova] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:02.446065 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14bc2ac-410d-477e-8ab4-2d7df9e94a41 service nova] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:02.451948 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c3d953-d160-41b8-9945-2d2e7bad34c8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:02.452499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c3d953-d160-41b8-9945-2d2e7bad34c8 service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:02.453425 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c3d953-d160-41b8-9945-2d2e7bad34c8 service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:02.453425 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c3d953-d160-41b8-9945-2d2e7bad34c8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:02.454948 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-e4c3d953-d160-41b8-9945-2d2e7bad34c8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:24:02.470052 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c3d953-d160-41b8-9945-2d2e7bad34c8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:02.471811 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c3d953-d160-41b8-9945-2d2e7bad34c8 service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:02.484058 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1374/2733] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:24:02 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:24:02.608017 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7791f0b-4ab2-4ccd-8482-d965cc622456 service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:02.610465 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-c7791f0b-4ab2-4ccd-8482-d965cc622456 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:24:02.644532 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:24:02.694704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7791f0b-4ab2-4ccd-8482-d965cc622456 service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:02.699252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e5a7c8-da90-4433-b05d-4dece43236c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:02.700498 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e5a7c8-da90-4433-b05d-4dece43236c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:02.700933 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e5a7c8-da90-4433-b05d-4dece43236c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:02.701528 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e5a7c8-da90-4433-b05d-4dece43236c5 admin admin] PATH_INFO: `/v3/users/6143ca6747fe481aa1cab63ed4897ed6` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:02.703060 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e5a7c8-da90-4433-b05d-4dece43236c5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6143ca6747fe481aa1cab63ed4897ed6)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:02.704879 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e5a7c8-da90-4433-b05d-4dece43236c5 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:02.759785 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d2e5a7c8-da90-4433-b05d-4dece43236c5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6143ca6747fe481aa1cab63ed4897ed6'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:02.760443 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d2e5a7c8-da90-4433-b05d-4dece43236c5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6143ca6747fe481aa1cab63ed4897ed6'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:02.767980 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d2e5a7c8-da90-4433-b05d-4dece43236c5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6143ca6747fe481aa1cab63ed4897ed6'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:02.782233 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d2e5a7c8-da90-4433-b05d-4dece43236c5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6143ca6747fe481aa1cab63ed4897ed6'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:02.782957 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d2e5a7c8-da90-4433-b05d-4dece43236c5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6143ca6747fe481aa1cab63ed4897ed6'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:02.799692 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1360/2734] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:24:02 2022] DELETE /identity/v3/users/6143ca6747fe481aa1cab63ed4897ed6 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:24:02.804175 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1a58ce-3dca-409a-9925-d54ea1077ad4 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:02.804531 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1a58ce-3dca-409a-9925-d54ea1077ad4 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:02.804869 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1a58ce-3dca-409a-9925-d54ea1077ad4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:02.813497 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:24:02.826310 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-7a1a58ce-3dca-409a-9925-d54ea1077ad4 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:24:02.828663 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-7a1a58ce-3dca-409a-9925-d54ea1077ad4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:24:02.894484 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1375/2735] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:24:02 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:24:02.905097 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e5a7c8-da90-4433-b05d-4dece43236c5 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:02.906716 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-d2e5a7c8-da90-4433-b05d-4dece43236c5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:24:02.971867 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e5a7c8-da90-4433-b05d-4dece43236c5 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:02.975788 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627a175c-9e39-438b-ae3b-008fb909816d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:02.976435 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627a175c-9e39-438b-ae3b-008fb909816d service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:02.976715 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627a175c-9e39-438b-ae3b-008fb909816d service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:02.977048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627a175c-9e39-438b-ae3b-008fb909816d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:02.989121 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627a175c-9e39-438b-ae3b-008fb909816d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:02.990152 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627a175c-9e39-438b-ae3b-008fb909816d service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:03.004855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1361/2736] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:24:02 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 16:24:03.317032 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1a58ce-3dca-409a-9925-d54ea1077ad4 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:03.318581 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-7a1a58ce-3dca-409a-9925-d54ea1077ad4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:24:03.333558 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1a58ce-3dca-409a-9925-d54ea1077ad4 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:03.338606 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecd7135-4cea-41ca-8733-866009d3df8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:03.339574 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecd7135-4cea-41ca-8733-866009d3df8d admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:03.339896 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecd7135-4cea-41ca-8733-866009d3df8d admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:03.340808 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecd7135-4cea-41ca-8733-866009d3df8d admin admin] PATH_INFO: `/v3/projects/7eb9a49bc8bf4fafa891c33ac3f7178d` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:03.342241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecd7135-4cea-41ca-8733-866009d3df8d admin admin] RBAC: Authorizing `identity:delete_project(project_id=7eb9a49bc8bf4fafa891c33ac3f7178d)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:03.344306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecd7135-4cea-41ca-8733-866009d3df8d admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:03.396728 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-fecd7135-4cea-41ca-8733-866009d3df8d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7eb9a49bc8bf4fafa891c33ac3f7178d', 'request_id': 'req-fecd7135-4cea-41ca-8733-866009d3df8d'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:03.404094 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-fecd7135-4cea-41ca-8733-866009d3df8d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7eb9a49bc8bf4fafa891c33ac3f7178d', 'request_id': 'req-fecd7135-4cea-41ca-8733-866009d3df8d'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:03.414228 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-fecd7135-4cea-41ca-8733-866009d3df8d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7eb9a49bc8bf4fafa891c33ac3f7178d', 'request_id': 'req-fecd7135-4cea-41ca-8733-866009d3df8d'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:03.428208 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-fecd7135-4cea-41ca-8733-866009d3df8d admin admin] The token cache is being invalidate because project 7eb9a49bc8bf4fafa891c33ac3f7178d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:24:03.428438 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-fecd7135-4cea-41ca-8733-866009d3df8d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:03.429057 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-fecd7135-4cea-41ca-8733-866009d3df8d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:03.430592 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1376/2737] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:24:03 2022] DELETE /identity/v3/projects/7eb9a49bc8bf4fafa891c33ac3f7178d => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:24:03.434882 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627a175c-9e39-438b-ae3b-008fb909816d service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:03.446128 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627a175c-9e39-438b-ae3b-008fb909816d service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:03.487891 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:24:03.507255 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a443b1-1cef-4c29-931b-a4c124b4cd2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:03.507922 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a443b1-1cef-4c29-931b-a4c124b4cd2a admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:03.508206 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a443b1-1cef-4c29-931b-a4c124b4cd2a admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:03.508502 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a443b1-1cef-4c29-931b-a4c124b4cd2a admin admin] PATH_INFO: `/v3/users/c5c20a34220040a1b03428d2dbd3f988` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:03.509622 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a443b1-1cef-4c29-931b-a4c124b4cd2a admin admin] RBAC: Authorizing `identity:delete_user(user_id=c5c20a34220040a1b03428d2dbd3f988)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:03.511576 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a443b1-1cef-4c29-931b-a4c124b4cd2a admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:03.563614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-03a443b1-1cef-4c29-931b-a4c124b4cd2a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c5c20a34220040a1b03428d2dbd3f988'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:03.564317 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-03a443b1-1cef-4c29-931b-a4c124b4cd2a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c5c20a34220040a1b03428d2dbd3f988'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:03.572251 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-03a443b1-1cef-4c29-931b-a4c124b4cd2a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c5c20a34220040a1b03428d2dbd3f988'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:03.585186 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-03a443b1-1cef-4c29-931b-a4c124b4cd2a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c5c20a34220040a1b03428d2dbd3f988'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:03.585912 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-03a443b1-1cef-4c29-931b-a4c124b4cd2a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c5c20a34220040a1b03428d2dbd3f988'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:03.600377 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1362/2738] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:24:03 2022] DELETE /identity/v3/users/c5c20a34220040a1b03428d2dbd3f988 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:24:03.890260 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecd7135-4cea-41ca-8733-866009d3df8d admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:03.891502 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-fecd7135-4cea-41ca-8733-866009d3df8d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:24:03.905142 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecd7135-4cea-41ca-8733-866009d3df8d admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:03.966810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1f0d7c-a496-4578-94d1-3d00bdf9a984 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:03.967441 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1f0d7c-a496-4578-94d1-3d00bdf9a984 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:03.967802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1f0d7c-a496-4578-94d1-3d00bdf9a984 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:03.968139 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1f0d7c-a496-4578-94d1-3d00bdf9a984 admin admin] PATH_INFO: `/v3/projects/61d01ca289c14b9987702695b5a41e73` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:03.969319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1f0d7c-a496-4578-94d1-3d00bdf9a984 admin admin] RBAC: Authorizing `identity:delete_project(project_id=61d01ca289c14b9987702695b5a41e73)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:03.971232 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1f0d7c-a496-4578-94d1-3d00bdf9a984 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:04.003021 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:24:04.019507 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-8d1f0d7c-a496-4578-94d1-3d00bdf9a984 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '61d01ca289c14b9987702695b5a41e73', 'request_id': 'req-8d1f0d7c-a496-4578-94d1-3d00bdf9a984'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:04.025553 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-8d1f0d7c-a496-4578-94d1-3d00bdf9a984 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '61d01ca289c14b9987702695b5a41e73', 'request_id': 'req-8d1f0d7c-a496-4578-94d1-3d00bdf9a984'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:04.031176 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-8d1f0d7c-a496-4578-94d1-3d00bdf9a984 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '61d01ca289c14b9987702695b5a41e73', 'request_id': 'req-8d1f0d7c-a496-4578-94d1-3d00bdf9a984'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:04.043171 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-8d1f0d7c-a496-4578-94d1-3d00bdf9a984 admin admin] The token cache is being invalidate because project 61d01ca289c14b9987702695b5a41e73 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:24:04.043464 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-8d1f0d7c-a496-4578-94d1-3d00bdf9a984 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:04.044672 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-8d1f0d7c-a496-4578-94d1-3d00bdf9a984 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:04.046154 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1377/2739] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:24:03 2022] DELETE /identity/v3/projects/61d01ca289c14b9987702695b5a41e73 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:24:04.052116 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da8e5d7-c592-4f10-9ec6-887dcb1da9fd None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:04.052590 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da8e5d7-c592-4f10-9ec6-887dcb1da9fd None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:04.052706 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da8e5d7-c592-4f10-9ec6-887dcb1da9fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:04.072777 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-6da8e5d7-c592-4f10-9ec6-887dcb1da9fd None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:24:04.073984 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-6da8e5d7-c592-4f10-9ec6-887dcb1da9fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:24:04.132951 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1363/2740] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:24:04 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:24:04.141201 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1199726b-3921-4364-9361-03fa0e43871b None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:04.141491 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1199726b-3921-4364-9361-03fa0e43871b None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:04.141772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1199726b-3921-4364-9361-03fa0e43871b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:04.161969 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-1199726b-3921-4364-9361-03fa0e43871b None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:24:04.163180 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-1199726b-3921-4364-9361-03fa0e43871b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:24:04.172053 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1378/2741] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:24:04 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:24:04.176302 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da8e5d7-c592-4f10-9ec6-887dcb1da9fd None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:04.186819 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da8e5d7-c592-4f10-9ec6-887dcb1da9fd None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:04.191444 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5ca84c-1b07-47b9-91ea-8cf460a7f946 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:04.192083 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5ca84c-1b07-47b9-91ea-8cf460a7f946 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:04.192366 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5ca84c-1b07-47b9-91ea-8cf460a7f946 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:04.192644 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5ca84c-1b07-47b9-91ea-8cf460a7f946 admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:04.193292 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e5ca84c-1b07-47b9-91ea-8cf460a7f946 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 16:24:04.193596 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e5ca84c-1b07-47b9-91ea-8cf460a7f946 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:04.195185 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e5ca84c-1b07-47b9-91ea-8cf460a7f946 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:04.206735 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1364/2742] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 16:24:04 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 16:24:04.211132 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1199726b-3921-4364-9361-03fa0e43871b None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:04.220742 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1199726b-3921-4364-9361-03fa0e43871b None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:04.225377 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02009bd-3762-4169-8fd4-01d7fc2a2529 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:04.225967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02009bd-3762-4169-8fd4-01d7fc2a2529 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:04.226251 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02009bd-3762-4169-8fd4-01d7fc2a2529 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:04.226522 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02009bd-3762-4169-8fd4-01d7fc2a2529 admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:04.227150 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02009bd-3762-4169-8fd4-01d7fc2a2529 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:04.229080 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02009bd-3762-4169-8fd4-01d7fc2a2529 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:04.256240 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1379/2743] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:24:04 2022] POST /identity/v3/projects => generated 376 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:24:04.260531 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5ca84c-1b07-47b9-91ea-8cf460a7f946 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:04.271301 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5ca84c-1b07-47b9-91ea-8cf460a7f946 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:04.317881 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:24:04.331561 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dcfea70-ac0e-4690-be99-1fc8f1da3361 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:04.332200 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcfea70-ac0e-4690-be99-1fc8f1da3361 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:04.332482 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcfea70-ac0e-4690-be99-1fc8f1da3361 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:04.332756 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcfea70-ac0e-4690-be99-1fc8f1da3361 admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:04.333399 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dcfea70-ac0e-4690-be99-1fc8f1da3361 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:04.335324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dcfea70-ac0e-4690-be99-1fc8f1da3361 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:04.358120 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1365/2744] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:24:04 2022] POST /identity/v3/users => generated 289 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 16:24:04.362512 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02009bd-3762-4169-8fd4-01d7fc2a2529 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:04.372779 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02009bd-3762-4169-8fd4-01d7fc2a2529 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:04.377282 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4509ff-5162-4e7a-a36b-86f03a04da7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:04.377872 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4509ff-5162-4e7a-a36b-86f03a04da7f admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:04.378147 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4509ff-5162-4e7a-a36b-86f03a04da7f admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:04.378413 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4509ff-5162-4e7a-a36b-86f03a04da7f admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:04.379049 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd4509ff-5162-4e7a-a36b-86f03a04da7f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:04.380731 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd4509ff-5162-4e7a-a36b-86f03a04da7f admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:04.391088 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1380/2745] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:24:04 2022] GET /identity/v3/roles => generated 1427 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:24:04.395164 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dcfea70-ac0e-4690-be99-1fc8f1da3361 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:04.406412 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dcfea70-ac0e-4690-be99-1fc8f1da3361 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:04.411203 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae11a13b-b49b-4295-95e1-466b28d95db2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:04.411830 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae11a13b-b49b-4295-95e1-466b28d95db2 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:04.412134 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae11a13b-b49b-4295-95e1-466b28d95db2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:04.412409 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae11a13b-b49b-4295-95e1-466b28d95db2 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:04.413048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae11a13b-b49b-4295-95e1-466b28d95db2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:04.414661 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae11a13b-b49b-4295-95e1-466b28d95db2 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:04.426053 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1366/2746] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:24:04 2022] GET /identity/v3/roles => generated 1427 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:24:04.430171 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4509ff-5162-4e7a-a36b-86f03a04da7f admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:04.440710 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4509ff-5162-4e7a-a36b-86f03a04da7f admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:04.445495 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d55aed9-083d-4c33-9dbf-853f5b0ca33c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:04.446144 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d55aed9-083d-4c33-9dbf-853f5b0ca33c admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:04.446422 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d55aed9-083d-4c33-9dbf-853f5b0ca33c admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:04.446692 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d55aed9-083d-4c33-9dbf-853f5b0ca33c admin admin] PATH_INFO: `/v3/projects/01cd3ebef4234a41ad268e262841257b/users/3d5090526c044d5f986189de3c169cd0/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:04.463481 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d55aed9-083d-4c33-9dbf-853f5b0ca33c admin admin] RBAC: Authorizing `identity:create_grant(project_id=01cd3ebef4234a41ad268e262841257b, user_id=3d5090526c044d5f986189de3c169cd0, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:04.467071 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d55aed9-083d-4c33-9dbf-853f5b0ca33c admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:04.480304 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1381/2747] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:24:04 2022] PUT /identity/v3/projects/01cd3ebef4234a41ad268e262841257b/users/3d5090526c044d5f986189de3c169cd0/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:24:04.491998 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3936849c-2112-4c52-8650-990ac9857f1a None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:04.492290 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3936849c-2112-4c52-8650-990ac9857f1a None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:04.492598 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3936849c-2112-4c52-8650-990ac9857f1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:04.521020 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-3936849c-2112-4c52-8650-990ac9857f1a None None] MFA Rules not processed for user `3d5090526c044d5f986189de3c169cd0`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:24:04.522246 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-3936849c-2112-4c52-8650-990ac9857f1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:24:04.586820 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1367/2748] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:24:04 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:24:04.598996 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d55aed9-083d-4c33-9dbf-853f5b0ca33c admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:04.600695 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-8d55aed9-083d-4c33-9dbf-853f5b0ca33c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:24:04.683647 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d55aed9-083d-4c33-9dbf-853f5b0ca33c admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:04.687710 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f7542a-1288-4ff9-a0bd-b58ae70fc7e8 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c6f06186c1a4626997db5a40c31cbb9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:04.688321 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f7542a-1288-4ff9-a0bd-b58ae70fc7e8 service glance] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:04.688605 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f7542a-1288-4ff9-a0bd-b58ae70fc7e8 service glance] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:04.688888 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f7542a-1288-4ff9-a0bd-b58ae70fc7e8 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:04.697925 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f7542a-1288-4ff9-a0bd-b58ae70fc7e8 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:04.699389 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f7542a-1288-4ff9-a0bd-b58ae70fc7e8 service glance] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:04.709882 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1382/2749] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:24:04 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:24:14.584016 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=44,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:24:14.705555 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=76,keystone:DELETE=5,keystone:INSERT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:24:22.324047 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3936849c-2112-4c52-8650-990ac9857f1a None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:22.389056 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3936849c-2112-4c52-8650-990ac9857f1a None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:22.394646 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08aaa55a-f481-436e-8421-296b91163c41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:22.395689 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08aaa55a-f481-436e-8421-296b91163c41 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:22.395979 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08aaa55a-f481-436e-8421-296b91163c41 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:22.396255 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08aaa55a-f481-436e-8421-296b91163c41 admin admin] PATH_INFO: `/v3/users/3d5090526c044d5f986189de3c169cd0` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:22.397432 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08aaa55a-f481-436e-8421-296b91163c41 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3d5090526c044d5f986189de3c169cd0)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:22.399367 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08aaa55a-f481-436e-8421-296b91163c41 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:22.447890 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-08aaa55a-f481-436e-8421-296b91163c41 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3d5090526c044d5f986189de3c169cd0'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:22.448541 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-08aaa55a-f481-436e-8421-296b91163c41 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3d5090526c044d5f986189de3c169cd0'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:22.456449 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-08aaa55a-f481-436e-8421-296b91163c41 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3d5090526c044d5f986189de3c169cd0'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:22.470283 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-08aaa55a-f481-436e-8421-296b91163c41 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3d5090526c044d5f986189de3c169cd0'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:22.471114 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-08aaa55a-f481-436e-8421-296b91163c41 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3d5090526c044d5f986189de3c169cd0'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:22.484089 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1368/2750] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:24:22 2022] DELETE /identity/v3/users/3d5090526c044d5f986189de3c169cd0 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:24:22.488664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4548e1a-feeb-40b0-8944-fe1bf4f2bc11 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:22.489237 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4548e1a-feeb-40b0-8944-fe1bf4f2bc11 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:22.489237 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4548e1a-feeb-40b0-8944-fe1bf4f2bc11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:22.507926 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-b4548e1a-feeb-40b0-8944-fe1bf4f2bc11 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:24:22.509203 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-b4548e1a-feeb-40b0-8944-fe1bf4f2bc11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:24:22.572936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1383/2751] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:24:22 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 16:24:22.583092 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08aaa55a-f481-436e-8421-296b91163c41 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:22.584525 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-08aaa55a-f481-436e-8421-296b91163c41 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:24:22.646968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08aaa55a-f481-436e-8421-296b91163c41 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:22.651128 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca508050-5cbd-491a-867b-ee025a802fa5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:22.651840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca508050-5cbd-491a-867b-ee025a802fa5 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:22.652145 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca508050-5cbd-491a-867b-ee025a802fa5 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:22.652444 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca508050-5cbd-491a-867b-ee025a802fa5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:22.662535 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca508050-5cbd-491a-867b-ee025a802fa5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:22.664033 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca508050-5cbd-491a-867b-ee025a802fa5 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:22.674031 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1369/2752] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:24:22 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:24:22.944857 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4548e1a-feeb-40b0-8944-fe1bf4f2bc11 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:22.946537 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-b4548e1a-feeb-40b0-8944-fe1bf4f2bc11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:24:22.959918 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4548e1a-feeb-40b0-8944-fe1bf4f2bc11 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:22.964785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83406479-d7d5-4122-8738-775c89217463 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:22.965420 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83406479-d7d5-4122-8738-775c89217463 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:22.965700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83406479-d7d5-4122-8738-775c89217463 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:22.965975 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83406479-d7d5-4122-8738-775c89217463 admin admin] PATH_INFO: `/v3/projects/01cd3ebef4234a41ad268e262841257b` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:22.968303 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83406479-d7d5-4122-8738-775c89217463 admin admin] RBAC: Authorizing `identity:delete_project(project_id=01cd3ebef4234a41ad268e262841257b)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:22.970235 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83406479-d7d5-4122-8738-775c89217463 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:23.019258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-83406479-d7d5-4122-8738-775c89217463 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '01cd3ebef4234a41ad268e262841257b', 'request_id': 'req-83406479-d7d5-4122-8738-775c89217463'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:23.025585 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-83406479-d7d5-4122-8738-775c89217463 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '01cd3ebef4234a41ad268e262841257b', 'request_id': 'req-83406479-d7d5-4122-8738-775c89217463'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:23.031944 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-83406479-d7d5-4122-8738-775c89217463 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '01cd3ebef4234a41ad268e262841257b', 'request_id': 'req-83406479-d7d5-4122-8738-775c89217463'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:23.045083 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-83406479-d7d5-4122-8738-775c89217463 admin admin] The token cache is being invalidate because project 01cd3ebef4234a41ad268e262841257b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:24:23.045402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-83406479-d7d5-4122-8738-775c89217463 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:23.046048 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-83406479-d7d5-4122-8738-775c89217463 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:23.047552 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1384/2753] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:24:22 2022] DELETE /identity/v3/projects/01cd3ebef4234a41ad268e262841257b => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:24:23.052517 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664df300-8857-4f23-949f-e8b24aa59293 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:23.052806 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664df300-8857-4f23-949f-e8b24aa59293 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:23.053083 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664df300-8857-4f23-949f-e8b24aa59293 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:23.074423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-664df300-8857-4f23-949f-e8b24aa59293 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:24:23.075548 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-664df300-8857-4f23-949f-e8b24aa59293 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:24:23.093977 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:24:23.137782 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1370/2754] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:24:23 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:24:23.146973 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af65f715-59a7-4819-a485-acced60a15a7 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:23.147257 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af65f715-59a7-4819-a485-acced60a15a7 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:23.147533 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af65f715-59a7-4819-a485-acced60a15a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:23.167344 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-af65f715-59a7-4819-a485-acced60a15a7 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:24:23.168574 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-af65f715-59a7-4819-a485-acced60a15a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:24:23.176336 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1385/2755] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:24:23 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:24:23.180502 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664df300-8857-4f23-949f-e8b24aa59293 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:23.190221 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664df300-8857-4f23-949f-e8b24aa59293 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:23.194807 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1d0716-6440-41e3-a3f8-24780fb466c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:23.195397 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1d0716-6440-41e3-a3f8-24780fb466c7 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:23.195702 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1d0716-6440-41e3-a3f8-24780fb466c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:23.195978 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1d0716-6440-41e3-a3f8-24780fb466c7 admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:23.196626 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1d0716-6440-41e3-a3f8-24780fb466c7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 16:24:23.196956 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1d0716-6440-41e3-a3f8-24780fb466c7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:23.198537 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1d0716-6440-41e3-a3f8-24780fb466c7 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:23.211374 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1371/2756] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 16:24:23 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 16:24:23.217125 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af65f715-59a7-4819-a485-acced60a15a7 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:23.226576 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af65f715-59a7-4819-a485-acced60a15a7 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:23.231481 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c601c6-7073-45b4-b6ac-4a22f960e1ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:23.232101 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c601c6-7073-45b4-b6ac-4a22f960e1ae admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:23.232380 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c601c6-7073-45b4-b6ac-4a22f960e1ae admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:23.232650 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c601c6-7073-45b4-b6ac-4a22f960e1ae admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:23.233287 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c601c6-7073-45b4-b6ac-4a22f960e1ae admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:23.235178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c601c6-7073-45b4-b6ac-4a22f960e1ae admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:23.265079 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1386/2757] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:24:23 2022] POST /identity/v3/projects => generated 378 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:24:23.269224 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1d0716-6440-41e3-a3f8-24780fb466c7 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:23.280578 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1d0716-6440-41e3-a3f8-24780fb466c7 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:23.337168 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00923b73-8fbc-4a75-ad3b-0f3ce43db5b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:23.337793 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00923b73-8fbc-4a75-ad3b-0f3ce43db5b3 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:23.338049 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00923b73-8fbc-4a75-ad3b-0f3ce43db5b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:23.338324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00923b73-8fbc-4a75-ad3b-0f3ce43db5b3 admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:23.339209 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00923b73-8fbc-4a75-ad3b-0f3ce43db5b3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:23.341135 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00923b73-8fbc-4a75-ad3b-0f3ce43db5b3 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:23.362852 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1372/2758] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:24:23 2022] POST /identity/v3/users => generated 290 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 16:24:23.367165 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c601c6-7073-45b4-b6ac-4a22f960e1ae admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:23.378237 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c601c6-7073-45b4-b6ac-4a22f960e1ae admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:23.382878 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ce0067-9763-4ae8-9ab6-3583ab33327d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:23.383476 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ce0067-9763-4ae8-9ab6-3583ab33327d admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:23.383800 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ce0067-9763-4ae8-9ab6-3583ab33327d admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:23.384074 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ce0067-9763-4ae8-9ab6-3583ab33327d admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:23.384722 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5ce0067-9763-4ae8-9ab6-3583ab33327d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:23.386312 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5ce0067-9763-4ae8-9ab6-3583ab33327d admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:23.396222 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1387/2759] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:24:23 2022] GET /identity/v3/roles => generated 1427 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:24:23.400136 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00923b73-8fbc-4a75-ad3b-0f3ce43db5b3 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:23.410483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00923b73-8fbc-4a75-ad3b-0f3ce43db5b3 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:23.415774 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f60beb3-5f7a-48e3-af75-cfdb5d4dc425 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:23.416400 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f60beb3-5f7a-48e3-af75-cfdb5d4dc425 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:23.416675 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f60beb3-5f7a-48e3-af75-cfdb5d4dc425 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:23.416949 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f60beb3-5f7a-48e3-af75-cfdb5d4dc425 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:23.417602 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f60beb3-5f7a-48e3-af75-cfdb5d4dc425 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:23.419208 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f60beb3-5f7a-48e3-af75-cfdb5d4dc425 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:23.430412 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1373/2760] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:24:23 2022] GET /identity/v3/roles => generated 1427 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 16:24:23.434518 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ce0067-9763-4ae8-9ab6-3583ab33327d admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:23.446266 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ce0067-9763-4ae8-9ab6-3583ab33327d admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:23.451390 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075289b0-5ebd-4027-ae82-3f5795396b3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:23.452138 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075289b0-5ebd-4027-ae82-3f5795396b3a admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:23.452424 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075289b0-5ebd-4027-ae82-3f5795396b3a admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:23.452702 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075289b0-5ebd-4027-ae82-3f5795396b3a admin admin] PATH_INFO: `/v3/projects/ebe2c62108544700b2f7146b3f799542/users/29730961b5564f37bceaa42ef0048249/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:23.469062 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075289b0-5ebd-4027-ae82-3f5795396b3a admin admin] RBAC: Authorizing `identity:create_grant(project_id=ebe2c62108544700b2f7146b3f799542, user_id=29730961b5564f37bceaa42ef0048249, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:23.471805 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075289b0-5ebd-4027-ae82-3f5795396b3a admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:23.487229 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1388/2761] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:24:23 2022] PUT /identity/v3/projects/ebe2c62108544700b2f7146b3f799542/users/29730961b5564f37bceaa42ef0048249/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:24:23.497270 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79bb9cd-01e8-404e-b0eb-352b6385c356 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:23.497571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79bb9cd-01e8-404e-b0eb-352b6385c356 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:23.498538 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79bb9cd-01e8-404e-b0eb-352b6385c356 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:23.527421 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-e79bb9cd-01e8-404e-b0eb-352b6385c356 None None] MFA Rules not processed for user `29730961b5564f37bceaa42ef0048249`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:24:23.528680 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-e79bb9cd-01e8-404e-b0eb-352b6385c356 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:24:23.596959 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1374/2762] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:24:23 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:24:23.608624 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075289b0-5ebd-4027-ae82-3f5795396b3a admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:23.610494 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-075289b0-5ebd-4027-ae82-3f5795396b3a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:24:23.631943 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:24:23.669037 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075289b0-5ebd-4027-ae82-3f5795396b3a admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:23.673358 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a526e0-eb23-480d-a34e-04e4e52903bc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:23.673964 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a526e0-eb23-480d-a34e-04e4e52903bc service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:23.674246 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a526e0-eb23-480d-a34e-04e4e52903bc service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:23.674522 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a526e0-eb23-480d-a34e-04e4e52903bc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:23.682985 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a526e0-eb23-480d-a34e-04e4e52903bc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:23.684453 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a526e0-eb23-480d-a34e-04e4e52903bc service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:23.694467 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1389/2763] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:24:23 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:24:28.276854 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79bb9cd-01e8-404e-b0eb-352b6385c356 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:28.278393 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-e79bb9cd-01e8-404e-b0eb-352b6385c356 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:24:28.309208 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:24:28.340693 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79bb9cd-01e8-404e-b0eb-352b6385c356 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:28.345583 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511a82c4-e904-4aef-95fe-ee778cba1d21 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:28.346318 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511a82c4-e904-4aef-95fe-ee778cba1d21 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:28.346671 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511a82c4-e904-4aef-95fe-ee778cba1d21 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:28.346983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511a82c4-e904-4aef-95fe-ee778cba1d21 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:28.348940 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-511a82c4-e904-4aef-95fe-ee778cba1d21 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:24:28.359466 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511a82c4-e904-4aef-95fe-ee778cba1d21 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:28.360921 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511a82c4-e904-4aef-95fe-ee778cba1d21 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:28.368913 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1375/2764] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:24:28 2022] GET /identity/v3/auth/tokens => generated 3131 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:24:33.690831 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=17 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:24:38.365211 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=18 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:24:44.362623 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a526e0-eb23-480d-a34e-04e4e52903bc service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:44.423197 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a526e0-eb23-480d-a34e-04e4e52903bc service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:44.428017 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d93ed1-d8ad-41be-ae62-5e25c1bca491 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:44.428651 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d93ed1-d8ad-41be-ae62-5e25c1bca491 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:44.428985 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d93ed1-d8ad-41be-ae62-5e25c1bca491 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:44.429293 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d93ed1-d8ad-41be-ae62-5e25c1bca491 admin admin] PATH_INFO: `/v3/users/29730961b5564f37bceaa42ef0048249` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:44.430422 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d93ed1-d8ad-41be-ae62-5e25c1bca491 admin admin] RBAC: Authorizing `identity:delete_user(user_id=29730961b5564f37bceaa42ef0048249)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:44.432443 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d93ed1-d8ad-41be-ae62-5e25c1bca491 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:44.480766 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-54d93ed1-d8ad-41be-ae62-5e25c1bca491 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '29730961b5564f37bceaa42ef0048249'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:44.481431 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-54d93ed1-d8ad-41be-ae62-5e25c1bca491 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '29730961b5564f37bceaa42ef0048249'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:44.482110 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-54d93ed1-d8ad-41be-ae62-5e25c1bca491 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '29730961b5564f37bceaa42ef0048249'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:44.489363 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-54d93ed1-d8ad-41be-ae62-5e25c1bca491 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '29730961b5564f37bceaa42ef0048249'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:44.501326 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-54d93ed1-d8ad-41be-ae62-5e25c1bca491 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '29730961b5564f37bceaa42ef0048249'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:44.513691 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1390/2765] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:24:44 2022] DELETE /identity/v3/users/29730961b5564f37bceaa42ef0048249 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:24:44.519367 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322e3a97-bebe-4194-b037-c158ef6c39ef None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:44.519898 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322e3a97-bebe-4194-b037-c158ef6c39ef None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:44.520014 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322e3a97-bebe-4194-b037-c158ef6c39ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:44.539043 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-322e3a97-bebe-4194-b037-c158ef6c39ef None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:24:44.540302 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-322e3a97-bebe-4194-b037-c158ef6c39ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:24:44.599068 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1376/2766] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:24:44 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:24:44.608880 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d93ed1-d8ad-41be-ae62-5e25c1bca491 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:44.610300 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-54d93ed1-d8ad-41be-ae62-5e25c1bca491 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:24:44.668081 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d93ed1-d8ad-41be-ae62-5e25c1bca491 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:44.671681 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6676bb-d21a-4ca2-9043-62c615a2eb33 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:44.672280 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6676bb-d21a-4ca2-9043-62c615a2eb33 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:44.672562 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6676bb-d21a-4ca2-9043-62c615a2eb33 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:44.672833 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6676bb-d21a-4ca2-9043-62c615a2eb33 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:44.682185 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f6676bb-d21a-4ca2-9043-62c615a2eb33 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:44.683615 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f6676bb-d21a-4ca2-9043-62c615a2eb33 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:44.699317 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1391/2767] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:24:44 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:24:44.977420 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322e3a97-bebe-4194-b037-c158ef6c39ef None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:44.978740 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-322e3a97-bebe-4194-b037-c158ef6c39ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:24:44.993514 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322e3a97-bebe-4194-b037-c158ef6c39ef None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:44.998878 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2925b58d-b790-4cbf-bdb6-ddc339792061 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:44.999555 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2925b58d-b790-4cbf-bdb6-ddc339792061 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:44.999811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2925b58d-b790-4cbf-bdb6-ddc339792061 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:45.000082 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2925b58d-b790-4cbf-bdb6-ddc339792061 admin admin] PATH_INFO: `/v3/projects/ebe2c62108544700b2f7146b3f799542` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:45.001371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2925b58d-b790-4cbf-bdb6-ddc339792061 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ebe2c62108544700b2f7146b3f799542)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:45.003917 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2925b58d-b790-4cbf-bdb6-ddc339792061 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:45.050899 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-2925b58d-b790-4cbf-bdb6-ddc339792061 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ebe2c62108544700b2f7146b3f799542', 'request_id': 'req-2925b58d-b790-4cbf-bdb6-ddc339792061'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:45.056987 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-2925b58d-b790-4cbf-bdb6-ddc339792061 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ebe2c62108544700b2f7146b3f799542', 'request_id': 'req-2925b58d-b790-4cbf-bdb6-ddc339792061'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:45.062469 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-2925b58d-b790-4cbf-bdb6-ddc339792061 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ebe2c62108544700b2f7146b3f799542', 'request_id': 'req-2925b58d-b790-4cbf-bdb6-ddc339792061'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:45.075558 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-2925b58d-b790-4cbf-bdb6-ddc339792061 admin admin] The token cache is being invalidate because project ebe2c62108544700b2f7146b3f799542 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:24:45.075886 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-2925b58d-b790-4cbf-bdb6-ddc339792061 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:45.076495 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-2925b58d-b790-4cbf-bdb6-ddc339792061 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:24:45.077968 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1377/2768] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:24:44 2022] DELETE /identity/v3/projects/ebe2c62108544700b2f7146b3f799542 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:24:45.083314 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7f7853-5cea-41e6-87a8-2fda362b5111 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:45.083606 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7f7853-5cea-41e6-87a8-2fda362b5111 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:45.083924 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7f7853-5cea-41e6-87a8-2fda362b5111 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:45.102935 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-6c7f7853-5cea-41e6-87a8-2fda362b5111 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:24:45.104144 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-6c7f7853-5cea-41e6-87a8-2fda362b5111 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:24:45.121149 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:24:45.186759 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1392/2769] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:24:45 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:24:45.195325 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb6f17d-b55e-46cd-9eb7-e4bb8f801676 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:45.195617 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb6f17d-b55e-46cd-9eb7-e4bb8f801676 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:45.195927 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb6f17d-b55e-46cd-9eb7-e4bb8f801676 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:45.216791 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-9bb6f17d-b55e-46cd-9eb7-e4bb8f801676 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:24:45.218068 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-9bb6f17d-b55e-46cd-9eb7-e4bb8f801676 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:24:45.226888 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1378/2770] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:24:45 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:24:45.231279 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7f7853-5cea-41e6-87a8-2fda362b5111 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:45.241861 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7f7853-5cea-41e6-87a8-2fda362b5111 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:45.246391 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fedcdb68-ea75-4518-8402-b86013db236c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:45.247011 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedcdb68-ea75-4518-8402-b86013db236c admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:45.247290 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedcdb68-ea75-4518-8402-b86013db236c admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:45.247564 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedcdb68-ea75-4518-8402-b86013db236c admin admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:45.248250 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fedcdb68-ea75-4518-8402-b86013db236c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 16:24:45.248557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fedcdb68-ea75-4518-8402-b86013db236c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:45.250153 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fedcdb68-ea75-4518-8402-b86013db236c admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:45.261993 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1393/2771] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 16:24:45 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 16:24:45.266278 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb6f17d-b55e-46cd-9eb7-e4bb8f801676 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:45.276233 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb6f17d-b55e-46cd-9eb7-e4bb8f801676 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:45.281884 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82854e1-cad0-4f5e-93cc-8c060026fe79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:45.282700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82854e1-cad0-4f5e-93cc-8c060026fe79 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:45.283552 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82854e1-cad0-4f5e-93cc-8c060026fe79 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:45.283859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82854e1-cad0-4f5e-93cc-8c060026fe79 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:45.284512 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a82854e1-cad0-4f5e-93cc-8c060026fe79 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:45.286468 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a82854e1-cad0-4f5e-93cc-8c060026fe79 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:45.315158 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1379/2772] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:24:45 2022] POST /identity/v3/projects => generated 372 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:24:45.319286 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fedcdb68-ea75-4518-8402-b86013db236c admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:45.329379 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fedcdb68-ea75-4518-8402-b86013db236c admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:45.387748 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f342ff97-1ae3-40b4-bdb8-eed4fa2bc1e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:45.388423 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f342ff97-1ae3-40b4-bdb8-eed4fa2bc1e9 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:45.388618 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f342ff97-1ae3-40b4-bdb8-eed4fa2bc1e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:45.388895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f342ff97-1ae3-40b4-bdb8-eed4fa2bc1e9 admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:45.389531 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f342ff97-1ae3-40b4-bdb8-eed4fa2bc1e9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:45.391454 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f342ff97-1ae3-40b4-bdb8-eed4fa2bc1e9 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:45.413414 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1394/2773] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:24:45 2022] POST /identity/v3/users => generated 287 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:24:45.417494 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82854e1-cad0-4f5e-93cc-8c060026fe79 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:45.429158 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82854e1-cad0-4f5e-93cc-8c060026fe79 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:45.433921 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1844cf4-cc59-4e30-bf8e-8dda898ae18d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:45.434617 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1844cf4-cc59-4e30-bf8e-8dda898ae18d admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:45.434859 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1844cf4-cc59-4e30-bf8e-8dda898ae18d admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:45.435167 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1844cf4-cc59-4e30-bf8e-8dda898ae18d admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:45.435834 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1844cf4-cc59-4e30-bf8e-8dda898ae18d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:45.437442 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1844cf4-cc59-4e30-bf8e-8dda898ae18d admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:45.448117 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1380/2774] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:24:45 2022] GET /identity/v3/roles => generated 1427 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:24:45.452088 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f342ff97-1ae3-40b4-bdb8-eed4fa2bc1e9 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:45.462850 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f342ff97-1ae3-40b4-bdb8-eed4fa2bc1e9 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:45.468146 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1b7012-9169-4525-9083-fb3910c1fd3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:45.468746 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1b7012-9169-4525-9083-fb3910c1fd3a admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:45.469028 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1b7012-9169-4525-9083-fb3910c1fd3a admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:45.469331 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1b7012-9169-4525-9083-fb3910c1fd3a admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:45.469977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1b7012-9169-4525-9083-fb3910c1fd3a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:45.472376 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1b7012-9169-4525-9083-fb3910c1fd3a admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:45.482892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1395/2775] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:24:45 2022] GET /identity/v3/roles => generated 1427 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 16:24:45.487697 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1844cf4-cc59-4e30-bf8e-8dda898ae18d admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:45.497811 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1844cf4-cc59-4e30-bf8e-8dda898ae18d admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:45.513218 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fffa93af-970e-432e-8d81-8a73279180a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:45.513906 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffa93af-970e-432e-8d81-8a73279180a3 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:45.514182 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffa93af-970e-432e-8d81-8a73279180a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:45.514456 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffa93af-970e-432e-8d81-8a73279180a3 admin admin] PATH_INFO: `/v3/projects/2d68315b538141f8b89af80186a5ff5c/users/e685baf05298436a95e9879edf5c91e4/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:45.532397 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fffa93af-970e-432e-8d81-8a73279180a3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2d68315b538141f8b89af80186a5ff5c, user_id=e685baf05298436a95e9879edf5c91e4, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:45.537589 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fffa93af-970e-432e-8d81-8a73279180a3 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:45.552593 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1381/2776] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:24:45 2022] PUT /identity/v3/projects/2d68315b538141f8b89af80186a5ff5c/users/e685baf05298436a95e9879edf5c91e4/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:24:45.557668 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25db6c37-06d9-4605-8580-b62fa4bcf676 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:45.557959 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25db6c37-06d9-4605-8580-b62fa4bcf676 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:45.558265 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25db6c37-06d9-4605-8580-b62fa4bcf676 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:45.576945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-25db6c37-06d9-4605-8580-b62fa4bcf676 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:24:45.578080 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-25db6c37-06d9-4605-8580-b62fa4bcf676 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:24:45.636241 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1396/2777] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:24:45 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:24:45.648603 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fffa93af-970e-432e-8d81-8a73279180a3 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:45.650324 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-fffa93af-970e-432e-8d81-8a73279180a3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:24:45.672847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:24:45.710876 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fffa93af-970e-432e-8d81-8a73279180a3 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:45.714818 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea494d8-0ff3-42a3-b866-32452afe1c53 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:45.715426 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea494d8-0ff3-42a3-b866-32452afe1c53 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:45.715745 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea494d8-0ff3-42a3-b866-32452afe1c53 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:45.716016 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea494d8-0ff3-42a3-b866-32452afe1c53 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:45.726216 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea494d8-0ff3-42a3-b866-32452afe1c53 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:45.727663 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea494d8-0ff3-42a3-b866-32452afe1c53 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:45.737561 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1382/2778] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:24:45 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:24:49.793065 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d230fe7-fed7-4cb2-84aa-6d8f56c5f583 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:49.793682 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d230fe7-fed7-4cb2-84aa-6d8f56c5f583 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:49.793682 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d230fe7-fed7-4cb2-84aa-6d8f56c5f583 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:49.823367 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-1d230fe7-fed7-4cb2-84aa-6d8f56c5f583 None None] MFA Rules not processed for user `e685baf05298436a95e9879edf5c91e4`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:24:49.824695 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-1d230fe7-fed7-4cb2-84aa-6d8f56c5f583 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:24:49.845754 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:24:49.903844 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1397/2779] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:24:49 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:24:49.908779 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea494d8-0ff3-42a3-b866-32452afe1c53 service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:49.919074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea494d8-0ff3-42a3-b866-32452afe1c53 service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:49.925017 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c541a911-185c-47cd-b6b6-4afcb681a7e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:49.925632 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c541a911-185c-47cd-b6b6-4afcb681a7e7 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:49.925911 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c541a911-185c-47cd-b6b6-4afcb681a7e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:49.926185 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c541a911-185c-47cd-b6b6-4afcb681a7e7 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:49.926825 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c541a911-185c-47cd-b6b6-4afcb681a7e7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:49.928788 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c541a911-185c-47cd-b6b6-4afcb681a7e7 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:49.958901 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1383/2780] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:24:49 2022] POST /identity/v3/projects => generated 372 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:24:49.963067 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d230fe7-fed7-4cb2-84aa-6d8f56c5f583 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:49.974970 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d230fe7-fed7-4cb2-84aa-6d8f56c5f583 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:50.034880 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b3025a-1d02-4a37-80f4-62d0ed77a2d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:50.035496 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b3025a-1d02-4a37-80f4-62d0ed77a2d1 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:50.035819 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b3025a-1d02-4a37-80f4-62d0ed77a2d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:50.036127 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b3025a-1d02-4a37-80f4-62d0ed77a2d1 admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:50.036781 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b3025a-1d02-4a37-80f4-62d0ed77a2d1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:50.038715 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b3025a-1d02-4a37-80f4-62d0ed77a2d1 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:50.068084 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1398/2781] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:24:49 2022] POST /identity/v3/users => generated 293 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:24:50.069653 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c541a911-185c-47cd-b6b6-4afcb681a7e7 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:50.080788 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c541a911-185c-47cd-b6b6-4afcb681a7e7 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:50.085531 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9374ea-5f81-44d8-b3fa-17c871c77943 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:50.086135 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9374ea-5f81-44d8-b3fa-17c871c77943 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:50.086417 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9374ea-5f81-44d8-b3fa-17c871c77943 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:50.086754 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9374ea-5f81-44d8-b3fa-17c871c77943 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:50.087403 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9374ea-5f81-44d8-b3fa-17c871c77943 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:50.089305 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9374ea-5f81-44d8-b3fa-17c871c77943 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:50.100699 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1384/2782] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:24:50 2022] GET /identity/v3/roles => generated 1427 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 16:24:50.104909 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b3025a-1d02-4a37-80f4-62d0ed77a2d1 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:50.116872 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b3025a-1d02-4a37-80f4-62d0ed77a2d1 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:50.121751 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20022cce-dc18-4391-9587-f7334442e2ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:50.122430 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20022cce-dc18-4391-9587-f7334442e2ad admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:50.122721 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20022cce-dc18-4391-9587-f7334442e2ad admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:50.122998 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20022cce-dc18-4391-9587-f7334442e2ad admin admin] PATH_INFO: `/v3/projects/a2b12163894040f8b03400616e97779f/users/ec93262317464cbfb6b8b38692b23349/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:50.140391 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20022cce-dc18-4391-9587-f7334442e2ad admin admin] RBAC: Authorizing `identity:create_grant(project_id=a2b12163894040f8b03400616e97779f, user_id=ec93262317464cbfb6b8b38692b23349, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:50.143322 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20022cce-dc18-4391-9587-f7334442e2ad admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:50.160213 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1399/2783] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:24:50 2022] PUT /identity/v3/projects/a2b12163894040f8b03400616e97779f/users/ec93262317464cbfb6b8b38692b23349/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:24:50.164128 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9374ea-5f81-44d8-b3fa-17c871c77943 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:50.175002 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9374ea-5f81-44d8-b3fa-17c871c77943 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:50.236795 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f138ef7c-d2d9-4950-81e9-e0d33b7e7717 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:50.237421 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f138ef7c-d2d9-4950-81e9-e0d33b7e7717 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:50.237704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f138ef7c-d2d9-4950-81e9-e0d33b7e7717 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:50.237976 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f138ef7c-d2d9-4950-81e9-e0d33b7e7717 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:50.238622 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f138ef7c-d2d9-4950-81e9-e0d33b7e7717 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:50.240282 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f138ef7c-d2d9-4950-81e9-e0d33b7e7717 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:50.250870 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1385/2784] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:24:50 2022] GET /identity/v3/roles => generated 1427 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:24:50.256188 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20022cce-dc18-4391-9587-f7334442e2ad admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:50.266899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20022cce-dc18-4391-9587-f7334442e2ad admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:50.271915 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2e1395-d444-4dd6-8b9a-d3790862abe4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:50.272570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2e1395-d444-4dd6-8b9a-d3790862abe4 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:50.272856 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2e1395-d444-4dd6-8b9a-d3790862abe4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:50.273142 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2e1395-d444-4dd6-8b9a-d3790862abe4 admin admin] PATH_INFO: `/v3/projects/a2b12163894040f8b03400616e97779f/users/ec93262317464cbfb6b8b38692b23349/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:50.275085 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b2e1395-d444-4dd6-8b9a-d3790862abe4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a2b12163894040f8b03400616e97779f, user_id=ec93262317464cbfb6b8b38692b23349, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:50.277777 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b2e1395-d444-4dd6-8b9a-d3790862abe4 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:50.289462 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1400/2785] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:24:50 2022] PUT /identity/v3/projects/a2b12163894040f8b03400616e97779f/users/ec93262317464cbfb6b8b38692b23349/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:24:54.947699 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7345545c-01f4-4572-bf33-20bf29e7d471 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:54.948341 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7345545c-01f4-4572-bf33-20bf29e7d471 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:54.948341 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7345545c-01f4-4572-bf33-20bf29e7d471 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:54.984158 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-7345545c-01f4-4572-bf33-20bf29e7d471 None None] MFA Rules not processed for user `ec93262317464cbfb6b8b38692b23349`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:24:54.985881 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-7345545c-01f4-4572-bf33-20bf29e7d471 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:24:55.069710 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1386/2786] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:24:54 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:24:55.083765 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2e1395-d444-4dd6-8b9a-d3790862abe4 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:24:55.085693 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-5b2e1395-d444-4dd6-8b9a-d3790862abe4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:24:55.166912 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2e1395-d444-4dd6-8b9a-d3790862abe4 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:24:55.172508 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8ce8c3-fe26-47cd-a3c0-43697fbbbfe1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:24:55.173834 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8ce8c3-fe26-47cd-a3c0-43697fbbbfe1 service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:55.174119 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8ce8c3-fe26-47cd-a3c0-43697fbbbfe1 service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:55.174382 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8ce8c3-fe26-47cd-a3c0-43697fbbbfe1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:55.180068 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:24:55.237384 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8ce8c3-fe26-47cd-a3c0-43697fbbbfe1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:24:55.239001 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8ce8c3-fe26-47cd-a3c0-43697fbbbfe1 service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:24:55.248747 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1401/2787] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:24:55 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:24:56.273257 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d7a9af-2fac-47f5-8674-4b3521e6dc97 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:24:56.273772 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d7a9af-2fac-47f5-8674-4b3521e6dc97 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:24:56.274187 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d7a9af-2fac-47f5-8674-4b3521e6dc97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:24:56.302960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-50d7a9af-2fac-47f5-8674-4b3521e6dc97 None None] MFA Rules not processed for user `e685baf05298436a95e9879edf5c91e4`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:24:56.304304 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-50d7a9af-2fac-47f5-8674-4b3521e6dc97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:24:56.311315 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1387/2788] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:24:56 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:25:05.244478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=24 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:25:06.297445 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=97,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:25:13.990775 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8ce8c3-fe26-47cd-a3c0-43697fbbbfe1 service nova] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:25:13.992317 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-cd8ce8c3-fe26-47cd-a3c0-43697fbbbfe1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:25:14.048918 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8ce8c3-fe26-47cd-a3c0-43697fbbbfe1 service nova] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:25:14.052576 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85f64a1-540e-47d1-93dc-418c1d2b4b92 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:25:14.053181 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85f64a1-540e-47d1-93dc-418c1d2b4b92 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:25:14.053459 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85f64a1-540e-47d1-93dc-418c1d2b4b92 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:25:14.053732 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85f64a1-540e-47d1-93dc-418c1d2b4b92 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:25:14.055570 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-c85f64a1-540e-47d1-93dc-418c1d2b4b92 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:25:14.065403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c85f64a1-540e-47d1-93dc-418c1d2b4b92 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:25:14.066841 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c85f64a1-540e-47d1-93dc-418c1d2b4b92 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:25:14.074869 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1402/2789] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:25:13 2022] GET /identity/v3/auth/tokens => generated 3131 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:25:24.071258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=26 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:25:31.433677 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d7a9af-2fac-47f5-8674-4b3521e6dc97 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:25:31.445533 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d7a9af-2fac-47f5-8674-4b3521e6dc97 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:25:31.449541 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90a6d96-53e7-473c-a237-2944564315b6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:25:31.450144 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90a6d96-53e7-473c-a237-2944564315b6 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:25:31.450423 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90a6d96-53e7-473c-a237-2944564315b6 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:25:31.450699 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90a6d96-53e7-473c-a237-2944564315b6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:25:31.461214 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c90a6d96-53e7-473c-a237-2944564315b6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:25:31.462650 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c90a6d96-53e7-473c-a237-2944564315b6 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:25:31.473280 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1388/2790] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:25:31 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:25:41.468444 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=6 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:26:15.180164 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a1e072-8adc-4502-bf1f-c55d0a432441 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:26:15.181517 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a1e072-8adc-4502-bf1f-c55d0a432441 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:26:15.182261 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a1e072-8adc-4502-bf1f-c55d0a432441 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:26:15.207194 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-79a1e072-8adc-4502-bf1f-c55d0a432441 None None] MFA Rules not processed for user `fe64f798929444dd838825dc0d2700cf`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:26:15.208808 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-79a1e072-8adc-4502-bf1f-c55d0a432441 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:26:15.275892 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1403/2791] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:26:15 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:26:25.268091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=25 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:26:25.527776 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90a6d96-53e7-473c-a237-2944564315b6 service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:26:25.529197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-c90a6d96-53e7-473c-a237-2944564315b6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:26:25.637028 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90a6d96-53e7-473c-a237-2944564315b6 service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:26:25.642231 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b35ad4-0fe4-45d9-b39d-8e1cde145da7 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0ee0002a76c3471f819b37fef510ef99', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:26:25.642833 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b35ad4-0fe4-45d9-b39d-8e1cde145da7 service placement] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:26:25.643154 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b35ad4-0fe4-45d9-b39d-8e1cde145da7 service placement] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:26:25.643601 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b35ad4-0fe4-45d9-b39d-8e1cde145da7 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:26:25.645375 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-e5b35ad4-0fe4-45d9-b39d-8e1cde145da7 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:26:25.656820 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b35ad4-0fe4-45d9-b39d-8e1cde145da7 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:26:25.658396 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b35ad4-0fe4-45d9-b39d-8e1cde145da7 service placement] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:26:25.668798 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1389/2792] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:26:25 2022] GET /identity/v3/auth/tokens => generated 3318 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:26:35.664570 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=48 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:27:30.697305 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a1e072-8adc-4502-bf1f-c55d0a432441 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:27:30.707845 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a1e072-8adc-4502-bf1f-c55d0a432441 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:27:30.711474 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5bdae5-b6f7-4ff1-bc33-3ef42422941a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:27:30.713326 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5bdae5-b6f7-4ff1-bc33-3ef42422941a service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:27:30.713602 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5bdae5-b6f7-4ff1-bc33-3ef42422941a service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:27:30.713876 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5bdae5-b6f7-4ff1-bc33-3ef42422941a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:27:30.715894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-3b5bdae5-b6f7-4ff1-bc33-3ef42422941a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:27:30.739353 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5bdae5-b6f7-4ff1-bc33-3ef42422941a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:27:30.740802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5bdae5-b6f7-4ff1-bc33-3ef42422941a service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:27:30.752117 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1404/2793] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:27:30 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:27:31.939424 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b35ad4-0fe4-45d9-b39d-8e1cde145da7 service placement] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:27:31.949617 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b35ad4-0fe4-45d9-b39d-8e1cde145da7 service placement] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:27:31.953520 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760097b9-1066-440c-9173-2080d637ded7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:27:31.954143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760097b9-1066-440c-9173-2080d637ded7 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:27:31.954424 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760097b9-1066-440c-9173-2080d637ded7 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:27:31.954694 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760097b9-1066-440c-9173-2080d637ded7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:27:31.956473 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-760097b9-1066-440c-9173-2080d637ded7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:27:31.971089 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-760097b9-1066-440c-9173-2080d637ded7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:27:31.972570 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-760097b9-1066-440c-9173-2080d637ded7 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:27:31.983788 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1390/2794] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:27:31 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:27:40.747455 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=10 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:27:41.978950 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=6 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:27:45.293668 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b60f299-3485-496c-ae6e-9a39e83b7e29 None None] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:27:45.293668 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b60f299-3485-496c-ae6e-9a39e83b7e29 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:27:45.293668 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b60f299-3485-496c-ae6e-9a39e83b7e29 None None] PATH_INFO: `/` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:27:45.295115 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1405/2795] 10.222.0.102 () {54 vars in 926 bytes} [Wed Aug 24 16:27:45 2022] GET /identity => generated 271 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 16:27:45.302640 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffacf4e-1bc4-40a1-b251-5b9ddb9d81ef None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:27:45.303297 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffacf4e-1bc4-40a1-b251-5b9ddb9d81ef None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:27:45.303297 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffacf4e-1bc4-40a1-b251-5b9ddb9d81ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:27:45.365343 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-6ffacf4e-1bc4-40a1-b251-5b9ddb9d81ef None None] MFA Rules not processed for user `48939674c1fd4cf599691d510fcd8858`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:27:45.366552 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-6ffacf4e-1bc4-40a1-b251-5b9ddb9d81ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:27:45.373474 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1391/2796] 10.222.0.102 () {58 vars in 1023 bytes} [Wed Aug 24 16:27:45 2022] POST /identity/v3/auth/tokens => generated 3131 bytes in 72 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:27:45.386488 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b60f299-3485-496c-ae6e-9a39e83b7e29 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:27:45.397557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b60f299-3485-496c-ae6e-9a39e83b7e29 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:27:45.401885 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7d056d-ff63-45e3-9425-d41c6bf3207f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:27:45.402479 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7d056d-ff63-45e3-9425-d41c6bf3207f service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:27:45.402758 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7d056d-ff63-45e3-9425-d41c6bf3207f service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:27:45.403030 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7d056d-ff63-45e3-9425-d41c6bf3207f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:27:45.410312 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7d056d-ff63-45e3-9425-d41c6bf3207f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:27:45.411782 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7d056d-ff63-45e3-9425-d41c6bf3207f service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:27:45.420038 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1406/2797] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:27:45 2022] GET /identity/v3/auth/tokens => generated 3131 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:27:45.503955 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5649ecd-321a-43ec-8a18-a578eaf2cc98 None None] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:27:45.504253 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5649ecd-321a-43ec-8a18-a578eaf2cc98 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:27:45.504531 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5649ecd-321a-43ec-8a18-a578eaf2cc98 None None] PATH_INFO: `/` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:27:45.505882 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1392/2798] 10.222.0.102 () {54 vars in 926 bytes} [Wed Aug 24 16:27:45 2022] GET /identity => generated 271 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 235 bytes (1 switches on core 0) Aug 24 16:27:45.511482 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd38d05d-9c11-4028-a818-250fb6a394fe None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:27:45.511792 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd38d05d-9c11-4028-a818-250fb6a394fe None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:27:45.512068 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd38d05d-9c11-4028-a818-250fb6a394fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:27:45.550757 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-fd38d05d-9c11-4028-a818-250fb6a394fe None None] MFA Rules not processed for user `e0d544379fe44438ad10289b45124374`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:27:45.551998 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-fd38d05d-9c11-4028-a818-250fb6a394fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:27:45.562024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1407/2799] 10.222.0.102 () {58 vars in 1023 bytes} [Wed Aug 24 16:27:45 2022] POST /identity/v3/auth/tokens => generated 3382 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:27:45.573425 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5649ecd-321a-43ec-8a18-a578eaf2cc98 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:27:45.584257 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5649ecd-321a-43ec-8a18-a578eaf2cc98 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:27:45.587847 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a160f2-24a5-4862-a522-52e2e78436c8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:27:45.588438 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a160f2-24a5-4862-a522-52e2e78436c8 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:27:45.588697 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a160f2-24a5-4862-a522-52e2e78436c8 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:27:45.588969 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a160f2-24a5-4862-a522-52e2e78436c8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:27:45.600024 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a160f2-24a5-4862-a522-52e2e78436c8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:27:45.601452 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a160f2-24a5-4862-a522-52e2e78436c8 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:27:45.612484 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1393/2800] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:27:45 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:27:52.516625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd38d05d-9c11-4028-a818-250fb6a394fe None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:27:52.518114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-fd38d05d-9c11-4028-a818-250fb6a394fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:27:52.576913 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd38d05d-9c11-4028-a818-250fb6a394fe None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:27:52.580691 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb35aed-a6bf-44f5-a393-4cbe6cc3f2de service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:27:52.581291 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb35aed-a6bf-44f5-a393-4cbe6cc3f2de service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:27:52.581599 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb35aed-a6bf-44f5-a393-4cbe6cc3f2de service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:27:52.581873 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb35aed-a6bf-44f5-a393-4cbe6cc3f2de service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:27:52.583615 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-7eb35aed-a6bf-44f5-a393-4cbe6cc3f2de service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:27:52.666838 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb35aed-a6bf-44f5-a393-4cbe6cc3f2de service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:27:52.668871 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb35aed-a6bf-44f5-a393-4cbe6cc3f2de service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:27:52.678373 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1408/2801] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:27:52 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:27:55.607589 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=23 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:28:02.674027 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=74 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:28:13.346117 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a160f2-24a5-4862-a522-52e2e78436c8 service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:28:13.375333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a160f2-24a5-4862-a522-52e2e78436c8 service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:28:13.378973 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8cd18a1-75b9-4f3f-9994-360b2602659c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:28:13.379563 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cd18a1-75b9-4f3f-9994-360b2602659c service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:28:13.379870 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cd18a1-75b9-4f3f-9994-360b2602659c service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:28:13.380143 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cd18a1-75b9-4f3f-9994-360b2602659c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:28:13.381870 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-a8cd18a1-75b9-4f3f-9994-360b2602659c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:28:13.393947 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8cd18a1-75b9-4f3f-9994-360b2602659c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:28:13.395425 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8cd18a1-75b9-4f3f-9994-360b2602659c service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:28:13.404538 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1394/2802] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:28:13 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:28:21.290015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5abc95f-0a75-491e-82fb-5a9c4f471d63 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:28:21.290479 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5abc95f-0a75-491e-82fb-5a9c4f471d63 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:28:21.290572 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5abc95f-0a75-491e-82fb-5a9c4f471d63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:28:21.311840 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-b5abc95f-0a75-491e-82fb-5a9c4f471d63 None None] MFA Rules not processed for user `fe64f798929444dd838825dc0d2700cf`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:28:21.313063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-b5abc95f-0a75-491e-82fb-5a9c4f471d63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:28:21.320103 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1409/2803] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:28:21 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:28:23.400351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=15 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:28:31.306304 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:29:23.265802 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8cd18a1-75b9-4f3f-9994-360b2602659c service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:29:23.305625 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8cd18a1-75b9-4f3f-9994-360b2602659c service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:29:23.312302 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca48bcb3-b438-42fd-9399-8d3af65057ed service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:29:23.313072 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca48bcb3-b438-42fd-9399-8d3af65057ed service nova] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:29:23.313455 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca48bcb3-b438-42fd-9399-8d3af65057ed service nova] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:29:23.313790 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca48bcb3-b438-42fd-9399-8d3af65057ed service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:29:23.317220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-ca48bcb3-b438-42fd-9399-8d3af65057ed service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:29:23.330817 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca48bcb3-b438-42fd-9399-8d3af65057ed service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:29:23.331514 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca48bcb3-b438-42fd-9399-8d3af65057ed service nova] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:29:23.344781 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1395/2804] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:29:23 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:29:25.750078 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5abc95f-0a75-491e-82fb-5a9c4f471d63 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:29:25.763941 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5abc95f-0a75-491e-82fb-5a9c4f471d63 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:29:25.770778 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a970a9dd-e56a-481a-8906-06b79c4edfad service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d544379fe44438ad10289b45124374', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:29:25.771494 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a970a9dd-e56a-481a-8906-06b79c4edfad service nova] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:29:25.771820 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a970a9dd-e56a-481a-8906-06b79c4edfad service nova] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:29:25.772734 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a970a9dd-e56a-481a-8906-06b79c4edfad service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:29:25.774616 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-a970a9dd-e56a-481a-8906-06b79c4edfad service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:29:25.788647 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a970a9dd-e56a-481a-8906-06b79c4edfad service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:29:25.790408 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a970a9dd-e56a-481a-8906-06b79c4edfad service nova] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:29:25.807172 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1410/2805] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:29:25 2022] GET /identity/v3/auth/tokens => generated 3382 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:29:30.215932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca48bcb3-b438-42fd-9399-8d3af65057ed service nova] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:29:30.227983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca48bcb3-b438-42fd-9399-8d3af65057ed service nova] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:29:30.239181 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd9a54f-a51d-4b1f-8fc3-a6b4db6506ad service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0ee0002a76c3471f819b37fef510ef99', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:29:30.239828 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd9a54f-a51d-4b1f-8fc3-a6b4db6506ad service placement] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:29:30.240107 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd9a54f-a51d-4b1f-8fc3-a6b4db6506ad service placement] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:29:30.240375 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd9a54f-a51d-4b1f-8fc3-a6b4db6506ad service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:29:30.242433 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-8bd9a54f-a51d-4b1f-8fc3-a6b4db6506ad service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:29:30.255242 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd9a54f-a51d-4b1f-8fc3-a6b4db6506ad service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:29:30.256969 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd9a54f-a51d-4b1f-8fc3-a6b4db6506ad service placement] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:29:30.268439 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1396/2806] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:29:30 2022] GET /identity/v3/auth/tokens => generated 3318 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:29:33.239183 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a970a9dd-e56a-481a-8906-06b79c4edfad service nova] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:29:33.258756 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a970a9dd-e56a-481a-8906-06b79c4edfad service nova] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:29:33.263977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a3f7e6-3758-4d44-9fd2-3a7f0c8da141 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:29:33.265167 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a3f7e6-3758-4d44-9fd2-3a7f0c8da141 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:29:33.265852 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a3f7e6-3758-4d44-9fd2-3a7f0c8da141 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:29:33.266023 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a3f7e6-3758-4d44-9fd2-3a7f0c8da141 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:29:33.268228 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-81a3f7e6-3758-4d44-9fd2-3a7f0c8da141 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:29:33.278979 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a3f7e6-3758-4d44-9fd2-3a7f0c8da141 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:29:33.281235 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a3f7e6-3758-4d44-9fd2-3a7f0c8da141 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:29:33.290311 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1411/2807] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:29:33 2022] GET /identity/v3/auth/tokens => generated 3131 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:29:33.642480 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd9a54f-a51d-4b1f-8fc3-a6b4db6506ad service placement] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:29:33.711428 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd9a54f-a51d-4b1f-8fc3-a6b4db6506ad service placement] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:29:33.716130 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234ca988-71cd-478a-bba5-b591beedcfc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:29:33.717033 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234ca988-71cd-478a-bba5-b591beedcfc6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:29:33.717311 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234ca988-71cd-478a-bba5-b591beedcfc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:29:33.717583 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234ca988-71cd-478a-bba5-b591beedcfc6 admin admin] PATH_INFO: `/v3/users/e685baf05298436a95e9879edf5c91e4` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:29:33.718736 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234ca988-71cd-478a-bba5-b591beedcfc6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e685baf05298436a95e9879edf5c91e4)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:29:33.720695 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234ca988-71cd-478a-bba5-b591beedcfc6 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:29:33.772194 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-234ca988-71cd-478a-bba5-b591beedcfc6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e685baf05298436a95e9879edf5c91e4'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:29:33.772931 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-234ca988-71cd-478a-bba5-b591beedcfc6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e685baf05298436a95e9879edf5c91e4'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:29:33.781090 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-234ca988-71cd-478a-bba5-b591beedcfc6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e685baf05298436a95e9879edf5c91e4'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:29:33.792893 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-234ca988-71cd-478a-bba5-b591beedcfc6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e685baf05298436a95e9879edf5c91e4'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:29:33.793604 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-234ca988-71cd-478a-bba5-b591beedcfc6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e685baf05298436a95e9879edf5c91e4'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:29:33.809732 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1397/2808] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:29:33 2022] DELETE /identity/v3/users/e685baf05298436a95e9879edf5c91e4 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:29:33.990486 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a3f7e6-3758-4d44-9fd2-3a7f0c8da141 service neutron] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:29:34.009147 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a3f7e6-3758-4d44-9fd2-3a7f0c8da141 service neutron] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:29:34.068861 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e5b3b6-706c-4141-907d-8a18e398951e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:29:34.069499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e5b3b6-706c-4141-907d-8a18e398951e admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:29:34.069775 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e5b3b6-706c-4141-907d-8a18e398951e admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:29:34.070055 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e5b3b6-706c-4141-907d-8a18e398951e admin admin] PATH_INFO: `/v3/projects/2d68315b538141f8b89af80186a5ff5c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:29:34.071219 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e5b3b6-706c-4141-907d-8a18e398951e admin admin] RBAC: Authorizing `identity:delete_project(project_id=2d68315b538141f8b89af80186a5ff5c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:29:34.073156 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e5b3b6-706c-4141-907d-8a18e398951e admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:29:34.121086 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-d1e5b3b6-706c-4141-907d-8a18e398951e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2d68315b538141f8b89af80186a5ff5c', 'request_id': 'req-d1e5b3b6-706c-4141-907d-8a18e398951e'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:29:34.127173 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-d1e5b3b6-706c-4141-907d-8a18e398951e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2d68315b538141f8b89af80186a5ff5c', 'request_id': 'req-d1e5b3b6-706c-4141-907d-8a18e398951e'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:29:34.133024 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-d1e5b3b6-706c-4141-907d-8a18e398951e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2d68315b538141f8b89af80186a5ff5c', 'request_id': 'req-d1e5b3b6-706c-4141-907d-8a18e398951e'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:29:34.145905 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-d1e5b3b6-706c-4141-907d-8a18e398951e admin admin] The token cache is being invalidate because project 2d68315b538141f8b89af80186a5ff5c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:29:34.146194 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-d1e5b3b6-706c-4141-907d-8a18e398951e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:29:34.146810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-d1e5b3b6-706c-4141-907d-8a18e398951e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:29:34.148378 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1412/2809] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:29:33 2022] DELETE /identity/v3/projects/2d68315b538141f8b89af80186a5ff5c => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:29:34.152454 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234ca988-71cd-478a-bba5-b591beedcfc6 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:29:34.153848 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-234ca988-71cd-478a-bba5-b591beedcfc6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:29:34.167186 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234ca988-71cd-478a-bba5-b591beedcfc6 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:29:34.201616 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:29:34.233186 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e151ebe7-cd5f-4aaf-b27f-ab6fd940d442 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:29:34.233850 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e151ebe7-cd5f-4aaf-b27f-ab6fd940d442 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:29:34.234172 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e151ebe7-cd5f-4aaf-b27f-ab6fd940d442 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:29:34.234490 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e151ebe7-cd5f-4aaf-b27f-ab6fd940d442 admin admin] PATH_INFO: `/v3/users/ec93262317464cbfb6b8b38692b23349` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:29:34.235909 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e151ebe7-cd5f-4aaf-b27f-ab6fd940d442 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ec93262317464cbfb6b8b38692b23349)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:29:34.237835 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e151ebe7-cd5f-4aaf-b27f-ab6fd940d442 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:29:34.283287 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-e151ebe7-cd5f-4aaf-b27f-ab6fd940d442 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ec93262317464cbfb6b8b38692b23349'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:29:34.283964 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-e151ebe7-cd5f-4aaf-b27f-ab6fd940d442 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ec93262317464cbfb6b8b38692b23349'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:29:34.291541 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-e151ebe7-cd5f-4aaf-b27f-ab6fd940d442 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ec93262317464cbfb6b8b38692b23349'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:29:34.304459 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-e151ebe7-cd5f-4aaf-b27f-ab6fd940d442 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ec93262317464cbfb6b8b38692b23349'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:29:34.305229 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-e151ebe7-cd5f-4aaf-b27f-ab6fd940d442 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ec93262317464cbfb6b8b38692b23349'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:29:34.317801 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1398/2810] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:29:34 2022] DELETE /identity/v3/users/ec93262317464cbfb6b8b38692b23349 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:29:34.496556 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e5b3b6-706c-4141-907d-8a18e398951e admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:29:34.498839 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-d1e5b3b6-706c-4141-907d-8a18e398951e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:29:34.512752 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e5b3b6-706c-4141-907d-8a18e398951e admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:29:34.573917 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:29:34.583472 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e045bef-5a97-452e-be7d-c2b8c55349e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:29:34.584400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e045bef-5a97-452e-be7d-c2b8c55349e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:29:34.584635 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e045bef-5a97-452e-be7d-c2b8c55349e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:29:34.584864 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e045bef-5a97-452e-be7d-c2b8c55349e8 admin admin] PATH_INFO: `/v3/projects/a2b12163894040f8b03400616e97779f` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:29:34.586382 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e045bef-5a97-452e-be7d-c2b8c55349e8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a2b12163894040f8b03400616e97779f)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:29:34.588556 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e045bef-5a97-452e-be7d-c2b8c55349e8 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:29:34.638380 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-9e045bef-5a97-452e-be7d-c2b8c55349e8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a2b12163894040f8b03400616e97779f', 'request_id': 'req-9e045bef-5a97-452e-be7d-c2b8c55349e8'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:29:34.645063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-9e045bef-5a97-452e-be7d-c2b8c55349e8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a2b12163894040f8b03400616e97779f', 'request_id': 'req-9e045bef-5a97-452e-be7d-c2b8c55349e8'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:29:34.650508 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-9e045bef-5a97-452e-be7d-c2b8c55349e8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a2b12163894040f8b03400616e97779f', 'request_id': 'req-9e045bef-5a97-452e-be7d-c2b8c55349e8'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:29:34.665893 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-9e045bef-5a97-452e-be7d-c2b8c55349e8 admin admin] The token cache is being invalidate because project a2b12163894040f8b03400616e97779f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:29:34.666187 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-9e045bef-5a97-452e-be7d-c2b8c55349e8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:29:34.666802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-9e045bef-5a97-452e-be7d-c2b8c55349e8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:29:34.668368 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1413/2811] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:29:34 2022] DELETE /identity/v3/projects/a2b12163894040f8b03400616e97779f => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:29:44.314309 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=36,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:29:44.662771 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=26,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:30:20.223264 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e151ebe7-cd5f-4aaf-b27f-ab6fd940d442 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:30:20.224788 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-e151ebe7-cd5f-4aaf-b27f-ab6fd940d442 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:30:20.282074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e151ebe7-cd5f-4aaf-b27f-ab6fd940d442 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:30:20.330875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c107e234-a5fe-4fda-b13f-03c7de56ec76 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:30:20.331483 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c107e234-a5fe-4fda-b13f-03c7de56ec76 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:30:20.331797 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c107e234-a5fe-4fda-b13f-03c7de56ec76 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:30:20.332067 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c107e234-a5fe-4fda-b13f-03c7de56ec76 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:30:20.333832 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-c107e234-a5fe-4fda-b13f-03c7de56ec76 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:30:20.400358 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c107e234-a5fe-4fda-b13f-03c7de56ec76 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:30:20.401797 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c107e234-a5fe-4fda-b13f-03c7de56ec76 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:30:20.412056 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1399/2812] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:30:20 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 189 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:30:26.821924 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e045bef-5a97-452e-be7d-c2b8c55349e8 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:30:26.823499 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-9e045bef-5a97-452e-be7d-c2b8c55349e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:30:26.837503 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e045bef-5a97-452e-be7d-c2b8c55349e8 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:30:26.842061 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac28c0b1-66c4-4617-a754-1e80fcc4b2c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:30:26.842940 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac28c0b1-66c4-4617-a754-1e80fcc4b2c7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:30:26.843219 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac28c0b1-66c4-4617-a754-1e80fcc4b2c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:30:26.843519 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac28c0b1-66c4-4617-a754-1e80fcc4b2c7 admin admin] PATH_INFO: `/v3/users/fe64f798929444dd838825dc0d2700cf` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:30:26.844733 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac28c0b1-66c4-4617-a754-1e80fcc4b2c7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fe64f798929444dd838825dc0d2700cf)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:30:26.847293 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac28c0b1-66c4-4617-a754-1e80fcc4b2c7 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:30:26.901291 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-ac28c0b1-66c4-4617-a754-1e80fcc4b2c7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fe64f798929444dd838825dc0d2700cf'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:30:26.902160 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-ac28c0b1-66c4-4617-a754-1e80fcc4b2c7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fe64f798929444dd838825dc0d2700cf'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:30:26.903181 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-ac28c0b1-66c4-4617-a754-1e80fcc4b2c7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fe64f798929444dd838825dc0d2700cf'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:30:26.921789 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-ac28c0b1-66c4-4617-a754-1e80fcc4b2c7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fe64f798929444dd838825dc0d2700cf'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:30:26.935555 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-ac28c0b1-66c4-4617-a754-1e80fcc4b2c7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fe64f798929444dd838825dc0d2700cf'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:30:26.949063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1414/2813] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:30:26 2022] DELETE /identity/v3/users/fe64f798929444dd838825dc0d2700cf => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:30:27.123133 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c107e234-a5fe-4fda-b13f-03c7de56ec76 service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:30:27.136651 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c107e234-a5fe-4fda-b13f-03c7de56ec76 service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:30:27.197654 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a82b893-ef58-464e-9663-e67bfe1dbfc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:30:27.198289 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a82b893-ef58-464e-9663-e67bfe1dbfc5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:30:27.198569 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a82b893-ef58-464e-9663-e67bfe1dbfc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:30:27.198844 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a82b893-ef58-464e-9663-e67bfe1dbfc5 admin admin] PATH_INFO: `/v3/projects/1259ac84ae764120b41de4af4b0cdb56` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:30:27.200005 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a82b893-ef58-464e-9663-e67bfe1dbfc5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1259ac84ae764120b41de4af4b0cdb56)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:30:27.201918 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a82b893-ef58-464e-9663-e67bfe1dbfc5 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:30:27.211095 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=100 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:30:27.250513 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-4a82b893-ef58-464e-9663-e67bfe1dbfc5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1259ac84ae764120b41de4af4b0cdb56', 'request_id': 'req-4a82b893-ef58-464e-9663-e67bfe1dbfc5'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:30:27.257444 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-4a82b893-ef58-464e-9663-e67bfe1dbfc5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1259ac84ae764120b41de4af4b0cdb56', 'request_id': 'req-4a82b893-ef58-464e-9663-e67bfe1dbfc5'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:30:27.262823 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-4a82b893-ef58-464e-9663-e67bfe1dbfc5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1259ac84ae764120b41de4af4b0cdb56', 'request_id': 'req-4a82b893-ef58-464e-9663-e67bfe1dbfc5'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:30:27.275929 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-4a82b893-ef58-464e-9663-e67bfe1dbfc5 admin admin] The token cache is being invalidate because project 1259ac84ae764120b41de4af4b0cdb56 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:30:27.276219 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-4a82b893-ef58-464e-9663-e67bfe1dbfc5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:30:27.276840 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-4a82b893-ef58-464e-9663-e67bfe1dbfc5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:30:27.278401 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1400/2814] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:30:27 2022] DELETE /identity/v3/projects/1259ac84ae764120b41de4af4b0cdb56 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:30:27.283256 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac28c0b1-66c4-4617-a754-1e80fcc4b2c7 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:30:27.284623 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-ac28c0b1-66c4-4617-a754-1e80fcc4b2c7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:30:27.297899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac28c0b1-66c4-4617-a754-1e80fcc4b2c7 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:30:27.358021 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8931656-510a-495e-8ea4-a30fd67f47ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:30:27.358661 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8931656-510a-495e-8ea4-a30fd67f47ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:30:27.358946 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8931656-510a-495e-8ea4-a30fd67f47ea admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:30:27.359221 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8931656-510a-495e-8ea4-a30fd67f47ea admin admin] PATH_INFO: `/v3/users/1956cb5416904bf4b30243e1318249d9` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:30:27.360995 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8931656-510a-495e-8ea4-a30fd67f47ea admin admin] RBAC: Authorizing `identity:delete_user(user_id=1956cb5416904bf4b30243e1318249d9)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:30:27.363572 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8931656-510a-495e-8ea4-a30fd67f47ea admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:30:27.411809 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-f8931656-510a-495e-8ea4-a30fd67f47ea admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1956cb5416904bf4b30243e1318249d9'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:30:27.412479 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-f8931656-510a-495e-8ea4-a30fd67f47ea admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1956cb5416904bf4b30243e1318249d9'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:30:27.413093 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-f8931656-510a-495e-8ea4-a30fd67f47ea admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1956cb5416904bf4b30243e1318249d9'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:30:27.420369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-f8931656-510a-495e-8ea4-a30fd67f47ea admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1956cb5416904bf4b30243e1318249d9'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:30:27.432786 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-f8931656-510a-495e-8ea4-a30fd67f47ea admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1956cb5416904bf4b30243e1318249d9'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:30:27.434702 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:30:27.447478 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1415/2815] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:30:27 2022] DELETE /identity/v3/users/1956cb5416904bf4b30243e1318249d9 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:30:27.617734 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a82b893-ef58-464e-9663-e67bfe1dbfc5 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:30:27.619583 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-4a82b893-ef58-464e-9663-e67bfe1dbfc5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:30:27.635736 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a82b893-ef58-464e-9663-e67bfe1dbfc5 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:30:27.694087 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b9ee57-322a-40e0-be74-5efdb66a2a1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:30:27.694773 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b9ee57-322a-40e0-be74-5efdb66a2a1f admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:30:27.695026 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b9ee57-322a-40e0-be74-5efdb66a2a1f admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:30:27.695375 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b9ee57-322a-40e0-be74-5efdb66a2a1f admin admin] PATH_INFO: `/v3/projects/c9935dd42b694ec68fef78a45b54709c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:30:27.696552 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b9ee57-322a-40e0-be74-5efdb66a2a1f admin admin] RBAC: Authorizing `identity:delete_project(project_id=c9935dd42b694ec68fef78a45b54709c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:30:27.698452 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b9ee57-322a-40e0-be74-5efdb66a2a1f admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:30:27.746936 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-a9b9ee57-322a-40e0-be74-5efdb66a2a1f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c9935dd42b694ec68fef78a45b54709c', 'request_id': 'req-a9b9ee57-322a-40e0-be74-5efdb66a2a1f'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:30:27.753268 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-a9b9ee57-322a-40e0-be74-5efdb66a2a1f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c9935dd42b694ec68fef78a45b54709c', 'request_id': 'req-a9b9ee57-322a-40e0-be74-5efdb66a2a1f'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:30:27.758762 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-a9b9ee57-322a-40e0-be74-5efdb66a2a1f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c9935dd42b694ec68fef78a45b54709c', 'request_id': 'req-a9b9ee57-322a-40e0-be74-5efdb66a2a1f'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:30:27.771083 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-a9b9ee57-322a-40e0-be74-5efdb66a2a1f admin admin] The token cache is being invalidate because project c9935dd42b694ec68fef78a45b54709c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:30:27.771369 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-a9b9ee57-322a-40e0-be74-5efdb66a2a1f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:30:27.772013 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-a9b9ee57-322a-40e0-be74-5efdb66a2a1f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:30:27.774827 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1401/2816] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:30:27 2022] DELETE /identity/v3/projects/c9935dd42b694ec68fef78a45b54709c => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:30:37.448502 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=4 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:30:37.768163 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=72,keystone:DELETE=14,keystone:INSERT=2 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:30:48.617069 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8931656-510a-495e-8ea4-a30fd67f47ea admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:30:48.618565 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-f8931656-510a-495e-8ea4-a30fd67f47ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:30:48.676459 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8931656-510a-495e-8ea4-a30fd67f47ea admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:30:48.730435 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07889dd1-a30b-4dd2-b3f4-01cd087bb59a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:30:48.730953 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07889dd1-a30b-4dd2-b3f4-01cd087bb59a service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:30:48.731230 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07889dd1-a30b-4dd2-b3f4-01cd087bb59a service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:30:48.731498 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07889dd1-a30b-4dd2-b3f4-01cd087bb59a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:30:48.733300 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-07889dd1-a30b-4dd2-b3f4-01cd087bb59a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:30:48.797516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07889dd1-a30b-4dd2-b3f4-01cd087bb59a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:30:48.800184 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07889dd1-a30b-4dd2-b3f4-01cd087bb59a service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:30:48.809967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1416/2817] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:30:48 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 193 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:30:51.676809 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b9ee57-322a-40e0-be74-5efdb66a2a1f admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:30:51.678272 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-a9b9ee57-322a-40e0-be74-5efdb66a2a1f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:30:51.751527 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07889dd1-a30b-4dd2-b3f4-01cd087bb59a service neutron] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:30:51.754042 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-07889dd1-a30b-4dd2-b3f4-01cd087bb59a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:30:51.768511 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b9ee57-322a-40e0-be74-5efdb66a2a1f admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:30:51.770950 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07889dd1-a30b-4dd2-b3f4-01cd087bb59a service neutron] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:30:51.773763 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02d1414-9356-4b9a-a4c8-1bc12bcb7f14 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0ee0002a76c3471f819b37fef510ef99', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:30:51.774668 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02d1414-9356-4b9a-a4c8-1bc12bcb7f14 service placement] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:30:51.775252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02d1414-9356-4b9a-a4c8-1bc12bcb7f14 service placement] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:30:51.775731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02d1414-9356-4b9a-a4c8-1bc12bcb7f14 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:30:51.776101 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b2a74f-2b5d-440b-8b95-109b702a59cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:30:51.776772 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b2a74f-2b5d-440b-8b95-109b702a59cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:30:51.777054 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b2a74f-2b5d-440b-8b95-109b702a59cd admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:30:51.777393 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b2a74f-2b5d-440b-8b95-109b702a59cd admin admin] PATH_INFO: `/v3/users/c058b07c1e9e443ebacbf38b1f13e58b` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:30:51.777781 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-e02d1414-9356-4b9a-a4c8-1bc12bcb7f14 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:30:51.778709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b2a74f-2b5d-440b-8b95-109b702a59cd admin admin] RBAC: Authorizing `identity:delete_user(user_id=c058b07c1e9e443ebacbf38b1f13e58b)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:30:51.780718 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b2a74f-2b5d-440b-8b95-109b702a59cd admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:30:51.792038 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e02d1414-9356-4b9a-a4c8-1bc12bcb7f14 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:30:51.793891 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e02d1414-9356-4b9a-a4c8-1bc12bcb7f14 service placement] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:30:51.805650 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1402/2818] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:30:51 2022] GET /identity/v3/auth/tokens => generated 3318 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 16:30:51.841308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-e7b2a74f-2b5d-440b-8b95-109b702a59cd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c058b07c1e9e443ebacbf38b1f13e58b'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:30:51.847272 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-e7b2a74f-2b5d-440b-8b95-109b702a59cd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c058b07c1e9e443ebacbf38b1f13e58b'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:30:51.848044 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-e7b2a74f-2b5d-440b-8b95-109b702a59cd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c058b07c1e9e443ebacbf38b1f13e58b'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:30:51.855978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-e7b2a74f-2b5d-440b-8b95-109b702a59cd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c058b07c1e9e443ebacbf38b1f13e58b'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:30:51.862871 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:30:51.871162 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-e7b2a74f-2b5d-440b-8b95-109b702a59cd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c058b07c1e9e443ebacbf38b1f13e58b'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:30:51.884518 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1417/2819] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:30:51 2022] DELETE /identity/v3/users/c058b07c1e9e443ebacbf38b1f13e58b => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:30:52.065421 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02d1414-9356-4b9a-a4c8-1bc12bcb7f14 service placement] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:30:52.076972 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02d1414-9356-4b9a-a4c8-1bc12bcb7f14 service placement] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:30:52.143408 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0294ec44-7122-443b-b08c-39168cfd67c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:30:52.144062 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0294ec44-7122-443b-b08c-39168cfd67c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:30:52.144345 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0294ec44-7122-443b-b08c-39168cfd67c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:30:52.144622 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0294ec44-7122-443b-b08c-39168cfd67c4 admin admin] PATH_INFO: `/v3/projects/0b5c7713b8574a09b7af9ab97b5ecbe1` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:30:52.145752 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0294ec44-7122-443b-b08c-39168cfd67c4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0b5c7713b8574a09b7af9ab97b5ecbe1)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:30:52.147682 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0294ec44-7122-443b-b08c-39168cfd67c4 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:30:52.193462 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-0294ec44-7122-443b-b08c-39168cfd67c4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0b5c7713b8574a09b7af9ab97b5ecbe1', 'request_id': 'req-0294ec44-7122-443b-b08c-39168cfd67c4'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:30:52.199406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-0294ec44-7122-443b-b08c-39168cfd67c4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0b5c7713b8574a09b7af9ab97b5ecbe1', 'request_id': 'req-0294ec44-7122-443b-b08c-39168cfd67c4'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:30:52.205376 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-0294ec44-7122-443b-b08c-39168cfd67c4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0b5c7713b8574a09b7af9ab97b5ecbe1', 'request_id': 'req-0294ec44-7122-443b-b08c-39168cfd67c4'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:30:52.218430 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-0294ec44-7122-443b-b08c-39168cfd67c4 admin admin] The token cache is being invalidate because project 0b5c7713b8574a09b7af9ab97b5ecbe1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:30:52.218720 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-0294ec44-7122-443b-b08c-39168cfd67c4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:30:52.219351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-0294ec44-7122-443b-b08c-39168cfd67c4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:30:52.221681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1403/2820] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:30:52 2022] DELETE /identity/v3/projects/0b5c7713b8574a09b7af9ab97b5ecbe1 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:30:52.227358 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23599fc9-cd45-4376-a99c-11e38c37ebae None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:30:52.227670 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23599fc9-cd45-4376-a99c-11e38c37ebae None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:30:52.227950 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23599fc9-cd45-4376-a99c-11e38c37ebae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:30:52.274780 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-23599fc9-cd45-4376-a99c-11e38c37ebae None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:30:52.276035 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-23599fc9-cd45-4376-a99c-11e38c37ebae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:30:52.337100 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1418/2821] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:30:52 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:30:52.347273 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcd1c70-3039-45f1-b2d4-4c94fc8e3e3d None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:30:52.347572 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcd1c70-3039-45f1-b2d4-4c94fc8e3e3d None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:30:52.347943 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcd1c70-3039-45f1-b2d4-4c94fc8e3e3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:30:52.367336 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-ffcd1c70-3039-45f1-b2d4-4c94fc8e3e3d None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:30:52.368582 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-ffcd1c70-3039-45f1-b2d4-4c94fc8e3e3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:30:52.377070 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1404/2822] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:30:52 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:30:52.381244 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23599fc9-cd45-4376-a99c-11e38c37ebae None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:30:52.390938 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23599fc9-cd45-4376-a99c-11e38c37ebae None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:30:52.396182 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebf9a2c-70c7-4884-8c1e-11d73287a0ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:30:52.396776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebf9a2c-70c7-4884-8c1e-11d73287a0ed admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:30:52.397081 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebf9a2c-70c7-4884-8c1e-11d73287a0ed admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:30:52.397354 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebf9a2c-70c7-4884-8c1e-11d73287a0ed admin admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:30:52.397999 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ebf9a2c-70c7-4884-8c1e-11d73287a0ed admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 16:30:52.398331 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ebf9a2c-70c7-4884-8c1e-11d73287a0ed admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:30:52.399986 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ebf9a2c-70c7-4884-8c1e-11d73287a0ed admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:30:52.412436 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1419/2823] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 16:30:52 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 16:30:52.416621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffcd1c70-3039-45f1-b2d4-4c94fc8e3e3d None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:30:52.425978 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffcd1c70-3039-45f1-b2d4-4c94fc8e3e3d None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:30:52.430744 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48ab3b8-a0e4-40b1-9a55-259dc690794d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:30:52.431365 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48ab3b8-a0e4-40b1-9a55-259dc690794d admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:30:52.431656 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48ab3b8-a0e4-40b1-9a55-259dc690794d admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:30:52.431971 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48ab3b8-a0e4-40b1-9a55-259dc690794d admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:30:52.432598 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c48ab3b8-a0e4-40b1-9a55-259dc690794d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:30:52.434499 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c48ab3b8-a0e4-40b1-9a55-259dc690794d admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:30:52.446447 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:30:52.473119 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1405/2824] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:30:52 2022] POST /identity/v3/projects => generated 368 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:30:52.477421 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebf9a2c-70c7-4884-8c1e-11d73287a0ed admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:30:52.487621 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebf9a2c-70c7-4884-8c1e-11d73287a0ed admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:30:52.554637 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5808226e-b180-461d-992c-17b6448bfd79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:30:52.555262 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5808226e-b180-461d-992c-17b6448bfd79 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:30:52.555543 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5808226e-b180-461d-992c-17b6448bfd79 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:30:52.555853 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5808226e-b180-461d-992c-17b6448bfd79 admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:30:52.556495 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5808226e-b180-461d-992c-17b6448bfd79 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:30:52.558429 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5808226e-b180-461d-992c-17b6448bfd79 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:30:52.581732 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1420/2825] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:30:52 2022] POST /identity/v3/users => generated 285 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:30:52.586034 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48ab3b8-a0e4-40b1-9a55-259dc690794d admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:30:52.598139 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48ab3b8-a0e4-40b1-9a55-259dc690794d admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:30:52.603500 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5f9bb2-8df0-40fb-b3f9-824f037cb358 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:30:52.604288 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5f9bb2-8df0-40fb-b3f9-824f037cb358 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:30:52.604577 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5f9bb2-8df0-40fb-b3f9-824f037cb358 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:30:52.604850 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5f9bb2-8df0-40fb-b3f9-824f037cb358 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:30:52.605654 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5f9bb2-8df0-40fb-b3f9-824f037cb358 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:30:52.607253 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5f9bb2-8df0-40fb-b3f9-824f037cb358 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:30:52.618592 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1406/2826] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:30:52 2022] GET /identity/v3/roles => generated 1427 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:30:52.621931 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5808226e-b180-461d-992c-17b6448bfd79 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:30:52.632779 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5808226e-b180-461d-992c-17b6448bfd79 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:30:52.637418 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1266a114-4b06-49b3-baf0-e305abb22415 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:30:52.638014 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1266a114-4b06-49b3-baf0-e305abb22415 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:30:52.638297 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1266a114-4b06-49b3-baf0-e305abb22415 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:30:52.638571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1266a114-4b06-49b3-baf0-e305abb22415 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:30:52.639221 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1266a114-4b06-49b3-baf0-e305abb22415 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:30:52.640851 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1266a114-4b06-49b3-baf0-e305abb22415 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:30:52.652105 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1421/2827] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:30:52 2022] GET /identity/v3/roles => generated 1427 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:30:52.656295 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5f9bb2-8df0-40fb-b3f9-824f037cb358 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:30:52.667029 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5f9bb2-8df0-40fb-b3f9-824f037cb358 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:30:52.672098 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b2312e-c9c2-4468-b4be-e53fa32f1b6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:30:52.672750 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b2312e-c9c2-4468-b4be-e53fa32f1b6a admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:30:52.673093 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b2312e-c9c2-4468-b4be-e53fa32f1b6a admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:30:52.673478 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b2312e-c9c2-4468-b4be-e53fa32f1b6a admin admin] PATH_INFO: `/v3/projects/becc69a93a3a4dd8a46ea65f882ba5b1/users/3cb1823f9c4244cda48e7790632c5177/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:30:52.688205 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b2312e-c9c2-4468-b4be-e53fa32f1b6a admin admin] RBAC: Authorizing `identity:create_grant(project_id=becc69a93a3a4dd8a46ea65f882ba5b1, user_id=3cb1823f9c4244cda48e7790632c5177, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:30:52.691444 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b2312e-c9c2-4468-b4be-e53fa32f1b6a admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:30:52.705351 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1407/2828] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:30:52 2022] PUT /identity/v3/projects/becc69a93a3a4dd8a46ea65f882ba5b1/users/3cb1823f9c4244cda48e7790632c5177/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:30:52.715444 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f1c5cf-20f1-4e09-bb7d-5368cdf9e47d None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:30:52.715759 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f1c5cf-20f1-4e09-bb7d-5368cdf9e47d None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:30:52.716078 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f1c5cf-20f1-4e09-bb7d-5368cdf9e47d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:30:52.743907 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-f2f1c5cf-20f1-4e09-bb7d-5368cdf9e47d None None] MFA Rules not processed for user `3cb1823f9c4244cda48e7790632c5177`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:30:52.745131 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-f2f1c5cf-20f1-4e09-bb7d-5368cdf9e47d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:30:52.754975 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:30:52.812739 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1422/2829] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:30:52 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:30:52.825511 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b2312e-c9c2-4468-b4be-e53fa32f1b6a admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:30:52.827278 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-b3b2312e-c9c2-4468-b4be-e53fa32f1b6a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:30:52.886519 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b2312e-c9c2-4468-b4be-e53fa32f1b6a admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:30:52.890189 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29cabbce-5f89-4af9-9f90-f59d473b225c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:30:52.890803 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cabbce-5f89-4af9-9f90-f59d473b225c service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:30:52.891068 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cabbce-5f89-4af9-9f90-f59d473b225c service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:30:52.891352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cabbce-5f89-4af9-9f90-f59d473b225c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:30:52.899836 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29cabbce-5f89-4af9-9f90-f59d473b225c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:30:52.901276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29cabbce-5f89-4af9-9f90-f59d473b225c service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:30:52.910161 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1408/2830] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:30:52 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:31:01.347721 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f1c5cf-20f1-4e09-bb7d-5368cdf9e47d None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:01.409951 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f1c5cf-20f1-4e09-bb7d-5368cdf9e47d None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:01.414647 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a009ed4-1b7a-4074-9f29-1f8e58c9984d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:01.415283 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a009ed4-1b7a-4074-9f29-1f8e58c9984d admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:01.415567 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a009ed4-1b7a-4074-9f29-1f8e58c9984d admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:01.415874 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a009ed4-1b7a-4074-9f29-1f8e58c9984d admin admin] PATH_INFO: `/v3/users/3cb1823f9c4244cda48e7790632c5177` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:01.417009 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a009ed4-1b7a-4074-9f29-1f8e58c9984d admin admin] RBAC: Authorizing `identity:delete_user(user_id=3cb1823f9c4244cda48e7790632c5177)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:01.418943 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a009ed4-1b7a-4074-9f29-1f8e58c9984d admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:01.465785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-2a009ed4-1b7a-4074-9f29-1f8e58c9984d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3cb1823f9c4244cda48e7790632c5177'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:01.466455 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-2a009ed4-1b7a-4074-9f29-1f8e58c9984d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3cb1823f9c4244cda48e7790632c5177'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:01.467057 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-2a009ed4-1b7a-4074-9f29-1f8e58c9984d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3cb1823f9c4244cda48e7790632c5177'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:01.474760 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-2a009ed4-1b7a-4074-9f29-1f8e58c9984d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3cb1823f9c4244cda48e7790632c5177'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:01.486991 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-2a009ed4-1b7a-4074-9f29-1f8e58c9984d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3cb1823f9c4244cda48e7790632c5177'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:01.499019 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1423/2831] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:31:01 2022] DELETE /identity/v3/users/3cb1823f9c4244cda48e7790632c5177 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:31:01.504109 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980ea9c9-5aa3-4167-9359-874b4ae57691 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:01.504622 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980ea9c9-5aa3-4167-9359-874b4ae57691 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:01.504724 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980ea9c9-5aa3-4167-9359-874b4ae57691 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:01.524057 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-980ea9c9-5aa3-4167-9359-874b4ae57691 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:31:01.525267 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-980ea9c9-5aa3-4167-9359-874b4ae57691 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:01.585282 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1409/2832] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:31:01 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:31:01.595422 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a009ed4-1b7a-4074-9f29-1f8e58c9984d admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:01.597438 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-2a009ed4-1b7a-4074-9f29-1f8e58c9984d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:01.631436 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:31:01.655973 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a009ed4-1b7a-4074-9f29-1f8e58c9984d admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:01.659781 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c688d0fa-3dc2-4a8c-9608-543abcbd8e29 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:01.660379 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c688d0fa-3dc2-4a8c-9608-543abcbd8e29 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:01.660658 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c688d0fa-3dc2-4a8c-9608-543abcbd8e29 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:01.660927 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c688d0fa-3dc2-4a8c-9608-543abcbd8e29 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:01.669796 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c688d0fa-3dc2-4a8c-9608-543abcbd8e29 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:01.671214 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c688d0fa-3dc2-4a8c-9608-543abcbd8e29 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:01.680773 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1424/2833] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:31:01 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:31:01.961258 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980ea9c9-5aa3-4167-9359-874b4ae57691 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:01.962732 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-980ea9c9-5aa3-4167-9359-874b4ae57691 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:01.975309 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980ea9c9-5aa3-4167-9359-874b4ae57691 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:01.980222 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc19aa28-c25e-4b92-9a8d-99724548c4da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:01.980927 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc19aa28-c25e-4b92-9a8d-99724548c4da admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:01.981284 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc19aa28-c25e-4b92-9a8d-99724548c4da admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:01.981627 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc19aa28-c25e-4b92-9a8d-99724548c4da admin admin] PATH_INFO: `/v3/projects/becc69a93a3a4dd8a46ea65f882ba5b1` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:01.982841 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc19aa28-c25e-4b92-9a8d-99724548c4da admin admin] RBAC: Authorizing `identity:delete_project(project_id=becc69a93a3a4dd8a46ea65f882ba5b1)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:01.985028 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc19aa28-c25e-4b92-9a8d-99724548c4da admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:02.031174 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=95,keystone:INSERT=2,keystone:DELETE=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:31:02.034545 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-fc19aa28-c25e-4b92-9a8d-99724548c4da admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'becc69a93a3a4dd8a46ea65f882ba5b1', 'request_id': 'req-fc19aa28-c25e-4b92-9a8d-99724548c4da'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:02.041801 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-fc19aa28-c25e-4b92-9a8d-99724548c4da admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'becc69a93a3a4dd8a46ea65f882ba5b1', 'request_id': 'req-fc19aa28-c25e-4b92-9a8d-99724548c4da'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:02.048068 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-fc19aa28-c25e-4b92-9a8d-99724548c4da admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'becc69a93a3a4dd8a46ea65f882ba5b1', 'request_id': 'req-fc19aa28-c25e-4b92-9a8d-99724548c4da'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:02.061291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-fc19aa28-c25e-4b92-9a8d-99724548c4da admin admin] The token cache is being invalidate because project becc69a93a3a4dd8a46ea65f882ba5b1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:31:02.061579 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-fc19aa28-c25e-4b92-9a8d-99724548c4da admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:02.062215 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-fc19aa28-c25e-4b92-9a8d-99724548c4da admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:02.063708 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1410/2834] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:31:01 2022] DELETE /identity/v3/projects/becc69a93a3a4dd8a46ea65f882ba5b1 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:31:02.069153 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3242f06a-d0f3-430d-8139-30d905cb0976 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:02.069467 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3242f06a-d0f3-430d-8139-30d905cb0976 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:02.069707 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3242f06a-d0f3-430d-8139-30d905cb0976 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:02.088607 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-3242f06a-d0f3-430d-8139-30d905cb0976 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:31:02.089743 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-3242f06a-d0f3-430d-8139-30d905cb0976 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:02.149446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1425/2835] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:31:02 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:31:02.157241 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5bfa67-e84c-4bc9-b78d-9fd22d8f6c45 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:02.157525 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5bfa67-e84c-4bc9-b78d-9fd22d8f6c45 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:02.157803 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5bfa67-e84c-4bc9-b78d-9fd22d8f6c45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:02.178211 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-9c5bfa67-e84c-4bc9-b78d-9fd22d8f6c45 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:31:02.179428 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-9c5bfa67-e84c-4bc9-b78d-9fd22d8f6c45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:02.187594 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1411/2836] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:31:02 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:31:02.191894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3242f06a-d0f3-430d-8139-30d905cb0976 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:02.201851 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3242f06a-d0f3-430d-8139-30d905cb0976 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:02.206819 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a87fad18-8e33-47a3-830f-4153e6cccff7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:02.207414 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87fad18-8e33-47a3-830f-4153e6cccff7 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:02.207727 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87fad18-8e33-47a3-830f-4153e6cccff7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:02.208006 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87fad18-8e33-47a3-830f-4153e6cccff7 admin admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:02.208658 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a87fad18-8e33-47a3-830f-4153e6cccff7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 16:31:02.208968 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a87fad18-8e33-47a3-830f-4153e6cccff7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:02.210571 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a87fad18-8e33-47a3-830f-4153e6cccff7 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:02.222885 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1426/2837] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 16:31:02 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 16:31:02.227156 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5bfa67-e84c-4bc9-b78d-9fd22d8f6c45 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:02.237172 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5bfa67-e84c-4bc9-b78d-9fd22d8f6c45 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:02.241848 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8c509c-611d-47c1-873e-956aea3a5c70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:02.242441 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8c509c-611d-47c1-873e-956aea3a5c70 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:02.242723 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8c509c-611d-47c1-873e-956aea3a5c70 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:02.242996 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8c509c-611d-47c1-873e-956aea3a5c70 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:02.243657 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8c509c-611d-47c1-873e-956aea3a5c70 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:02.245565 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8c509c-611d-47c1-873e-956aea3a5c70 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:02.273893 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1412/2838] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:31:02 2022] POST /identity/v3/projects => generated 380 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:31:02.277997 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a87fad18-8e33-47a3-830f-4153e6cccff7 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:02.288821 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a87fad18-8e33-47a3-830f-4153e6cccff7 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:02.346835 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a89938-1275-4302-9c57-6bf69897977b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:02.347367 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a89938-1275-4302-9c57-6bf69897977b admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:02.347680 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a89938-1275-4302-9c57-6bf69897977b admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:02.347962 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a89938-1275-4302-9c57-6bf69897977b admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:02.348606 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a89938-1275-4302-9c57-6bf69897977b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:02.350525 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a89938-1275-4302-9c57-6bf69897977b admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:02.372758 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1427/2839] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:31:02 2022] POST /identity/v3/users => generated 291 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:31:02.378893 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8c509c-611d-47c1-873e-956aea3a5c70 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:02.389396 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8c509c-611d-47c1-873e-956aea3a5c70 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:02.394367 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10800ced-04af-45ae-904e-36df702d5634 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:02.394956 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10800ced-04af-45ae-904e-36df702d5634 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:02.395319 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10800ced-04af-45ae-904e-36df702d5634 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:02.395531 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10800ced-04af-45ae-904e-36df702d5634 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:02.396207 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10800ced-04af-45ae-904e-36df702d5634 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:02.397793 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10800ced-04af-45ae-904e-36df702d5634 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:02.407603 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1413/2840] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:31:02 2022] GET /identity/v3/roles => generated 1427 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:31:02.412376 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a89938-1275-4302-9c57-6bf69897977b admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:02.422682 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a89938-1275-4302-9c57-6bf69897977b admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:02.428315 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fa8a1f-f301-4b71-b88a-46d251e3e409 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:02.429350 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fa8a1f-f301-4b71-b88a-46d251e3e409 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:02.429727 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fa8a1f-f301-4b71-b88a-46d251e3e409 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:02.430045 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fa8a1f-f301-4b71-b88a-46d251e3e409 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:02.430709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8fa8a1f-f301-4b71-b88a-46d251e3e409 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:02.432656 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8fa8a1f-f301-4b71-b88a-46d251e3e409 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:02.447387 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1428/2841] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:31:02 2022] GET /identity/v3/roles => generated 1427 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:31:02.452717 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10800ced-04af-45ae-904e-36df702d5634 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:02.463810 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10800ced-04af-45ae-904e-36df702d5634 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:02.468798 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ead8499-b015-432b-9ed1-df2b22599502 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:02.469446 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ead8499-b015-432b-9ed1-df2b22599502 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:02.469728 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ead8499-b015-432b-9ed1-df2b22599502 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:02.470001 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ead8499-b015-432b-9ed1-df2b22599502 admin admin] PATH_INFO: `/v3/projects/737c087488264704ae1838530da5d48e/users/ced9f5a6957c47aa8cd85b935761b444/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:02.485163 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ead8499-b015-432b-9ed1-df2b22599502 admin admin] RBAC: Authorizing `identity:create_grant(project_id=737c087488264704ae1838530da5d48e, user_id=ced9f5a6957c47aa8cd85b935761b444, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:02.488710 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ead8499-b015-432b-9ed1-df2b22599502 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:02.501665 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1414/2842] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:31:02 2022] PUT /identity/v3/projects/737c087488264704ae1838530da5d48e/users/ced9f5a6957c47aa8cd85b935761b444/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:31:02.511572 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ded5360-e66b-41d7-8536-b0c73fd93cca None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:02.511894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ded5360-e66b-41d7-8536-b0c73fd93cca None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:02.512175 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ded5360-e66b-41d7-8536-b0c73fd93cca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:02.542451 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-9ded5360-e66b-41d7-8536-b0c73fd93cca None None] MFA Rules not processed for user `ced9f5a6957c47aa8cd85b935761b444`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:31:02.543329 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-9ded5360-e66b-41d7-8536-b0c73fd93cca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:02.576879 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:31:02.616200 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1429/2843] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:31:02 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:31:02.620610 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ead8499-b015-432b-9ed1-df2b22599502 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:02.682217 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ead8499-b015-432b-9ed1-df2b22599502 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:02.686875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69308c5b-01e5-44f5-bf4a-003fdf6a8c49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:02.687472 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69308c5b-01e5-44f5-bf4a-003fdf6a8c49 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:02.687796 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69308c5b-01e5-44f5-bf4a-003fdf6a8c49 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:02.688072 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69308c5b-01e5-44f5-bf4a-003fdf6a8c49 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:02.688715 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69308c5b-01e5-44f5-bf4a-003fdf6a8c49 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:02.690652 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69308c5b-01e5-44f5-bf4a-003fdf6a8c49 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:02.718750 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1415/2844] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:31:02 2022] POST /identity/v3/projects => generated 380 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 16:31:02.724107 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ded5360-e66b-41d7-8536-b0c73fd93cca None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:02.735439 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ded5360-e66b-41d7-8536-b0c73fd93cca None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:02.794092 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f38ec1d-0433-4c9f-9692-e791a4208720 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:02.794700 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f38ec1d-0433-4c9f-9692-e791a4208720 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:02.794985 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f38ec1d-0433-4c9f-9692-e791a4208720 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:02.795263 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f38ec1d-0433-4c9f-9692-e791a4208720 admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:02.795936 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f38ec1d-0433-4c9f-9692-e791a4208720 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:02.797894 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f38ec1d-0433-4c9f-9692-e791a4208720 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:02.820782 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1430/2845] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:31:02 2022] POST /identity/v3/users => generated 297 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:31:02.824780 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69308c5b-01e5-44f5-bf4a-003fdf6a8c49 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:02.836063 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69308c5b-01e5-44f5-bf4a-003fdf6a8c49 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:02.840735 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f2de8a-3253-41e5-9d71-d304a73c7b02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:02.841337 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f2de8a-3253-41e5-9d71-d304a73c7b02 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:02.841621 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f2de8a-3253-41e5-9d71-d304a73c7b02 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:02.841897 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f2de8a-3253-41e5-9d71-d304a73c7b02 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:02.842546 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f2de8a-3253-41e5-9d71-d304a73c7b02 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:02.844449 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f2de8a-3253-41e5-9d71-d304a73c7b02 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:02.855739 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1416/2846] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:31:02 2022] GET /identity/v3/roles => generated 1427 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:31:02.859819 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f38ec1d-0433-4c9f-9692-e791a4208720 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:02.870126 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f38ec1d-0433-4c9f-9692-e791a4208720 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:02.874709 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06e6c15-b8e0-4ca3-a9d7-90836e6c2432 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:02.875359 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06e6c15-b8e0-4ca3-a9d7-90836e6c2432 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:02.875726 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06e6c15-b8e0-4ca3-a9d7-90836e6c2432 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:02.875963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06e6c15-b8e0-4ca3-a9d7-90836e6c2432 admin admin] PATH_INFO: `/v3/projects/7115e8db982a45a39b4a75a69ad0556b/users/f11f284f8e5e43b7bdf8941bc1b9608e/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:02.891178 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06e6c15-b8e0-4ca3-a9d7-90836e6c2432 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7115e8db982a45a39b4a75a69ad0556b, user_id=f11f284f8e5e43b7bdf8941bc1b9608e, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:02.894485 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06e6c15-b8e0-4ca3-a9d7-90836e6c2432 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:02.907420 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1431/2847] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:31:02 2022] PUT /identity/v3/projects/7115e8db982a45a39b4a75a69ad0556b/users/f11f284f8e5e43b7bdf8941bc1b9608e/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:31:02.911355 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f2de8a-3253-41e5-9d71-d304a73c7b02 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:02.922076 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f2de8a-3253-41e5-9d71-d304a73c7b02 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:02.964257 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:DELETE=4,keystone:SELECT=92,keystone:INSERT=4 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:31:02.980166 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2e1f92-5d5e-4ffc-b7ef-ec8633fdd329 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:02.980766 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2e1f92-5d5e-4ffc-b7ef-ec8633fdd329 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:02.981042 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2e1f92-5d5e-4ffc-b7ef-ec8633fdd329 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:02.981311 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2e1f92-5d5e-4ffc-b7ef-ec8633fdd329 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:02.981948 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2e1f92-5d5e-4ffc-b7ef-ec8633fdd329 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:02.983550 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2e1f92-5d5e-4ffc-b7ef-ec8633fdd329 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:02.993427 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1417/2848] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:31:02 2022] GET /identity/v3/roles => generated 1427 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:31:02.997535 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06e6c15-b8e0-4ca3-a9d7-90836e6c2432 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:03.008263 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06e6c15-b8e0-4ca3-a9d7-90836e6c2432 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:03.012842 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821c9104-5598-4649-877f-fbe38a4a6cfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:03.013473 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821c9104-5598-4649-877f-fbe38a4a6cfa admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:03.013749 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821c9104-5598-4649-877f-fbe38a4a6cfa admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:03.014045 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821c9104-5598-4649-877f-fbe38a4a6cfa admin admin] PATH_INFO: `/v3/projects/7115e8db982a45a39b4a75a69ad0556b/users/f11f284f8e5e43b7bdf8941bc1b9608e/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:03.015966 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-821c9104-5598-4649-877f-fbe38a4a6cfa admin admin] RBAC: Authorizing `identity:create_grant(project_id=7115e8db982a45a39b4a75a69ad0556b, user_id=f11f284f8e5e43b7bdf8941bc1b9608e, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:03.019388 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-821c9104-5598-4649-877f-fbe38a4a6cfa admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:03.029507 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1432/2849] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:31:02 2022] PUT /identity/v3/projects/7115e8db982a45a39b4a75a69ad0556b/users/f11f284f8e5e43b7bdf8941bc1b9608e/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:31:03.039430 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7505f9e3-c18a-482c-afa1-d16f600cc0de None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:03.039745 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7505f9e3-c18a-482c-afa1-d16f600cc0de None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:03.040024 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7505f9e3-c18a-482c-afa1-d16f600cc0de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:03.067898 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-7505f9e3-c18a-482c-afa1-d16f600cc0de None None] MFA Rules not processed for user `f11f284f8e5e43b7bdf8941bc1b9608e`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:31:03.069115 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-7505f9e3-c18a-482c-afa1-d16f600cc0de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:03.140866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1418/2850] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:31:03 2022] POST /identity/v3/auth/tokens => generated 3333 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:31:03.155048 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821c9104-5598-4649-877f-fbe38a4a6cfa admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:03.156829 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-821c9104-5598-4649-877f-fbe38a4a6cfa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:03.217809 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821c9104-5598-4649-877f-fbe38a4a6cfa admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:03.223052 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7087df69-7151-4cb3-ad66-b6bbce717da2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:03.223704 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7087df69-7151-4cb3-ad66-b6bbce717da2 service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:03.223992 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7087df69-7151-4cb3-ad66-b6bbce717da2 service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:03.224269 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7087df69-7151-4cb3-ad66-b6bbce717da2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:03.233647 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7087df69-7151-4cb3-ad66-b6bbce717da2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:03.235088 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7087df69-7151-4cb3-ad66-b6bbce717da2 service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:03.245755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1433/2851] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:31:03 2022] GET /identity/v3/auth/tokens => generated 3333 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:31:03.322438 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7505f9e3-c18a-482c-afa1-d16f600cc0de None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:03.333592 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7505f9e3-c18a-482c-afa1-d16f600cc0de None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:03.337264 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a36756-c3f0-46b2-99f4-080640cfb6b4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:03.337869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a36756-c3f0-46b2-99f4-080640cfb6b4 service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:03.338147 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a36756-c3f0-46b2-99f4-080640cfb6b4 service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:03.338418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a36756-c3f0-46b2-99f4-080640cfb6b4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:03.396068 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a36756-c3f0-46b2-99f4-080640cfb6b4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:03.397457 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a36756-c3f0-46b2-99f4-080640cfb6b4 service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:03.406214 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1419/2852] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:31:03 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:31:13.241417 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=84,keystone:INSERT=5 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:31:13.288145 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7087df69-7151-4cb3-ad66-b6bbce717da2 service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:13.354623 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7087df69-7151-4cb3-ad66-b6bbce717da2 service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:13.359121 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4628e3e9-2d27-4b5a-a540-9656b8436ee2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:13.359785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4628e3e9-2d27-4b5a-a540-9656b8436ee2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:13.360063 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4628e3e9-2d27-4b5a-a540-9656b8436ee2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:13.360332 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4628e3e9-2d27-4b5a-a540-9656b8436ee2 admin admin] PATH_INFO: `/v3/users/ced9f5a6957c47aa8cd85b935761b444` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:13.361468 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4628e3e9-2d27-4b5a-a540-9656b8436ee2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ced9f5a6957c47aa8cd85b935761b444)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:13.363994 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4628e3e9-2d27-4b5a-a540-9656b8436ee2 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:13.402183 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=71 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:31:13.410383 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4628e3e9-2d27-4b5a-a540-9656b8436ee2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ced9f5a6957c47aa8cd85b935761b444'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:13.411037 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4628e3e9-2d27-4b5a-a540-9656b8436ee2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ced9f5a6957c47aa8cd85b935761b444'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:13.411656 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4628e3e9-2d27-4b5a-a540-9656b8436ee2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ced9f5a6957c47aa8cd85b935761b444'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:13.419158 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4628e3e9-2d27-4b5a-a540-9656b8436ee2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ced9f5a6957c47aa8cd85b935761b444'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:13.432614 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4628e3e9-2d27-4b5a-a540-9656b8436ee2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ced9f5a6957c47aa8cd85b935761b444'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:13.450593 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1434/2853] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:31:13 2022] DELETE /identity/v3/users/ced9f5a6957c47aa8cd85b935761b444 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:31:13.455197 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947abb81-09e2-43b6-8ba9-d03b3c4baaa2 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:13.455484 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947abb81-09e2-43b6-8ba9-d03b3c4baaa2 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:13.455796 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947abb81-09e2-43b6-8ba9-d03b3c4baaa2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:13.476250 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-947abb81-09e2-43b6-8ba9-d03b3c4baaa2 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:31:13.477464 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-947abb81-09e2-43b6-8ba9-d03b3c4baaa2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:13.538277 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1420/2854] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:31:13 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:31:13.548322 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4628e3e9-2d27-4b5a-a540-9656b8436ee2 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:13.550656 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-4628e3e9-2d27-4b5a-a540-9656b8436ee2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:13.608861 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4628e3e9-2d27-4b5a-a540-9656b8436ee2 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:13.613201 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bec0ffe-d4d1-4836-84b9-118ac1bf4713 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:13.613804 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bec0ffe-d4d1-4836-84b9-118ac1bf4713 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:13.614081 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bec0ffe-d4d1-4836-84b9-118ac1bf4713 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:13.614354 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bec0ffe-d4d1-4836-84b9-118ac1bf4713 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:13.623566 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bec0ffe-d4d1-4836-84b9-118ac1bf4713 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:13.625091 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bec0ffe-d4d1-4836-84b9-118ac1bf4713 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:13.635052 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1435/2855] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:31:13 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:31:13.903920 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947abb81-09e2-43b6-8ba9-d03b3c4baaa2 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:13.905352 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-947abb81-09e2-43b6-8ba9-d03b3c4baaa2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:13.918559 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947abb81-09e2-43b6-8ba9-d03b3c4baaa2 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:13.923997 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de9bd4f-c71a-40bd-963f-b645b255c6ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:13.924679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de9bd4f-c71a-40bd-963f-b645b255c6ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:13.924975 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de9bd4f-c71a-40bd-963f-b645b255c6ed admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:13.925316 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de9bd4f-c71a-40bd-963f-b645b255c6ed admin admin] PATH_INFO: `/v3/projects/737c087488264704ae1838530da5d48e` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:13.926610 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de9bd4f-c71a-40bd-963f-b645b255c6ed admin admin] RBAC: Authorizing `identity:delete_project(project_id=737c087488264704ae1838530da5d48e)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:13.928578 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de9bd4f-c71a-40bd-963f-b645b255c6ed admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:13.977730 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-8de9bd4f-c71a-40bd-963f-b645b255c6ed admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '737c087488264704ae1838530da5d48e', 'request_id': 'req-8de9bd4f-c71a-40bd-963f-b645b255c6ed'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:13.983770 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-8de9bd4f-c71a-40bd-963f-b645b255c6ed admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '737c087488264704ae1838530da5d48e', 'request_id': 'req-8de9bd4f-c71a-40bd-963f-b645b255c6ed'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:13.989802 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-8de9bd4f-c71a-40bd-963f-b645b255c6ed admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '737c087488264704ae1838530da5d48e', 'request_id': 'req-8de9bd4f-c71a-40bd-963f-b645b255c6ed'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:14.002282 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-8de9bd4f-c71a-40bd-963f-b645b255c6ed admin admin] The token cache is being invalidate because project 737c087488264704ae1838530da5d48e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:31:14.002571 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-8de9bd4f-c71a-40bd-963f-b645b255c6ed admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:14.003900 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-8de9bd4f-c71a-40bd-963f-b645b255c6ed admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:14.005454 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1421/2856] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:31:13 2022] DELETE /identity/v3/projects/737c087488264704ae1838530da5d48e => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:31:14.009493 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bec0ffe-d4d1-4836-84b9-118ac1bf4713 service neutron] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:14.020271 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bec0ffe-d4d1-4836-84b9-118ac1bf4713 service neutron] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:14.040599 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:31:14.081909 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e82f02-fc88-497c-bae8-f6a783040d03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:14.082542 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e82f02-fc88-497c-bae8-f6a783040d03 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:14.082817 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e82f02-fc88-497c-bae8-f6a783040d03 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:14.083083 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e82f02-fc88-497c-bae8-f6a783040d03 admin admin] PATH_INFO: `/v3/users/f11f284f8e5e43b7bdf8941bc1b9608e` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:14.084247 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e82f02-fc88-497c-bae8-f6a783040d03 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f11f284f8e5e43b7bdf8941bc1b9608e)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:14.086230 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e82f02-fc88-497c-bae8-f6a783040d03 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:14.132146 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-e8e82f02-fc88-497c-bae8-f6a783040d03 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f11f284f8e5e43b7bdf8941bc1b9608e'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:14.132804 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-e8e82f02-fc88-497c-bae8-f6a783040d03 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f11f284f8e5e43b7bdf8941bc1b9608e'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:14.133409 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-e8e82f02-fc88-497c-bae8-f6a783040d03 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f11f284f8e5e43b7bdf8941bc1b9608e'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:14.145250 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-e8e82f02-fc88-497c-bae8-f6a783040d03 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f11f284f8e5e43b7bdf8941bc1b9608e'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:14.166483 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-e8e82f02-fc88-497c-bae8-f6a783040d03 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f11f284f8e5e43b7bdf8941bc1b9608e'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:14.179535 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1436/2857] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:31:14 2022] DELETE /identity/v3/users/f11f284f8e5e43b7bdf8941bc1b9608e => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:31:14.450991 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de9bd4f-c71a-40bd-963f-b645b255c6ed admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:14.452567 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-8de9bd4f-c71a-40bd-963f-b645b255c6ed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:14.475202 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de9bd4f-c71a-40bd-963f-b645b255c6ed admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:14.538162 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4106c099-ae5f-4456-af58-e5620eac7888 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:14.538722 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4106c099-ae5f-4456-af58-e5620eac7888 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:14.539001 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4106c099-ae5f-4456-af58-e5620eac7888 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:14.539284 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4106c099-ae5f-4456-af58-e5620eac7888 admin admin] PATH_INFO: `/v3/projects/7115e8db982a45a39b4a75a69ad0556b` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:14.540458 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4106c099-ae5f-4456-af58-e5620eac7888 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7115e8db982a45a39b4a75a69ad0556b)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:14.542370 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4106c099-ae5f-4456-af58-e5620eac7888 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:14.574089 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:31:14.592467 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-4106c099-ae5f-4456-af58-e5620eac7888 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7115e8db982a45a39b4a75a69ad0556b', 'request_id': 'req-4106c099-ae5f-4456-af58-e5620eac7888'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:14.600737 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-4106c099-ae5f-4456-af58-e5620eac7888 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7115e8db982a45a39b4a75a69ad0556b', 'request_id': 'req-4106c099-ae5f-4456-af58-e5620eac7888'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:14.606905 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-4106c099-ae5f-4456-af58-e5620eac7888 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7115e8db982a45a39b4a75a69ad0556b', 'request_id': 'req-4106c099-ae5f-4456-af58-e5620eac7888'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:14.621046 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-4106c099-ae5f-4456-af58-e5620eac7888 admin admin] The token cache is being invalidate because project 7115e8db982a45a39b4a75a69ad0556b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:31:14.621337 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-4106c099-ae5f-4456-af58-e5620eac7888 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:14.621987 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-4106c099-ae5f-4456-af58-e5620eac7888 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:14.623475 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1422/2858] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:31:14 2022] DELETE /identity/v3/projects/7115e8db982a45a39b4a75a69ad0556b => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:31:14.628510 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38258a3f-f25f-4e3a-84b5-489b0347ecd8 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:14.628998 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38258a3f-f25f-4e3a-84b5-489b0347ecd8 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:14.628998 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38258a3f-f25f-4e3a-84b5-489b0347ecd8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:14.649402 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-38258a3f-f25f-4e3a-84b5-489b0347ecd8 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:31:14.650608 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-38258a3f-f25f-4e3a-84b5-489b0347ecd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:14.710205 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1437/2859] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:31:14 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:31:14.719199 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24abbe1e-3759-44c7-b234-41a575c1c9da None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:14.719489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24abbe1e-3759-44c7-b234-41a575c1c9da None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:14.719878 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24abbe1e-3759-44c7-b234-41a575c1c9da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:14.741225 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-24abbe1e-3759-44c7-b234-41a575c1c9da None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:31:14.742440 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-24abbe1e-3759-44c7-b234-41a575c1c9da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:14.750478 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1423/2860] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:31:14 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:31:14.754811 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38258a3f-f25f-4e3a-84b5-489b0347ecd8 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:14.764625 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38258a3f-f25f-4e3a-84b5-489b0347ecd8 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:14.769890 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5882eba2-91be-4388-8ab7-a4195513a749 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:14.770490 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5882eba2-91be-4388-8ab7-a4195513a749 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:14.770775 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5882eba2-91be-4388-8ab7-a4195513a749 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:14.771051 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5882eba2-91be-4388-8ab7-a4195513a749 admin admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:14.771743 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5882eba2-91be-4388-8ab7-a4195513a749 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 16:31:14.772056 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5882eba2-91be-4388-8ab7-a4195513a749 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:14.773657 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5882eba2-91be-4388-8ab7-a4195513a749 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:14.786445 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1438/2861] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 16:31:14 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 16:31:14.790813 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24abbe1e-3759-44c7-b234-41a575c1c9da None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:14.800836 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24abbe1e-3759-44c7-b234-41a575c1c9da None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:14.805831 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97e06f1-56da-4f4f-898e-366b65c18d1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:14.806452 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97e06f1-56da-4f4f-898e-366b65c18d1d admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:14.806731 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97e06f1-56da-4f4f-898e-366b65c18d1d admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:14.807022 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97e06f1-56da-4f4f-898e-366b65c18d1d admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:14.807679 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97e06f1-56da-4f4f-898e-366b65c18d1d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:14.810304 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97e06f1-56da-4f4f-898e-366b65c18d1d admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:14.839024 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1424/2862] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:31:14 2022] POST /identity/v3/projects => generated 366 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:31:14.843476 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5882eba2-91be-4388-8ab7-a4195513a749 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:14.854369 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5882eba2-91be-4388-8ab7-a4195513a749 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:14.880664 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:31:14.914640 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6933d35a-f227-4bc5-ae52-bf6577457b8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:14.915266 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6933d35a-f227-4bc5-ae52-bf6577457b8f admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:14.915557 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6933d35a-f227-4bc5-ae52-bf6577457b8f admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:14.915860 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6933d35a-f227-4bc5-ae52-bf6577457b8f admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:14.916500 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6933d35a-f227-4bc5-ae52-bf6577457b8f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:14.918440 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6933d35a-f227-4bc5-ae52-bf6577457b8f admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:14.940426 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1439/2863] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:31:14 2022] POST /identity/v3/users => generated 284 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 16:31:14.944781 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97e06f1-56da-4f4f-898e-366b65c18d1d admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:14.955298 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97e06f1-56da-4f4f-898e-366b65c18d1d admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:14.960715 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e9abaf-f2f8-4478-a66c-a1702b65a3d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:14.961319 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e9abaf-f2f8-4478-a66c-a1702b65a3d4 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:14.961601 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e9abaf-f2f8-4478-a66c-a1702b65a3d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:14.961876 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e9abaf-f2f8-4478-a66c-a1702b65a3d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:14.962532 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e9abaf-f2f8-4478-a66c-a1702b65a3d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:14.964188 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e9abaf-f2f8-4478-a66c-a1702b65a3d4 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:14.975131 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1425/2864] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:31:14 2022] GET /identity/v3/roles => generated 1427 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 16:31:14.978913 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6933d35a-f227-4bc5-ae52-bf6577457b8f admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:14.990008 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6933d35a-f227-4bc5-ae52-bf6577457b8f admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:14.994693 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d832bf-2e89-4775-9f5c-7430e28c1d8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:14.995303 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d832bf-2e89-4775-9f5c-7430e28c1d8f admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:14.995587 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d832bf-2e89-4775-9f5c-7430e28c1d8f admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:14.995893 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d832bf-2e89-4775-9f5c-7430e28c1d8f admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:14.996541 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d832bf-2e89-4775-9f5c-7430e28c1d8f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:14.998132 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d832bf-2e89-4775-9f5c-7430e28c1d8f admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:15.011838 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1440/2865] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:31:14 2022] GET /identity/v3/roles => generated 1427 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:31:15.015932 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e9abaf-f2f8-4478-a66c-a1702b65a3d4 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:15.026420 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e9abaf-f2f8-4478-a66c-a1702b65a3d4 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:15.031097 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727ac4af-86d0-4115-9c86-07eb2417e78e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:15.031781 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727ac4af-86d0-4115-9c86-07eb2417e78e admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:15.032062 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727ac4af-86d0-4115-9c86-07eb2417e78e admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:15.032333 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727ac4af-86d0-4115-9c86-07eb2417e78e admin admin] PATH_INFO: `/v3/projects/168b55fa58274864a658fda1fdc82a04/users/0cc924e2dc8843c2873f0aba5e7ef5c5/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:15.047692 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727ac4af-86d0-4115-9c86-07eb2417e78e admin admin] RBAC: Authorizing `identity:create_grant(project_id=168b55fa58274864a658fda1fdc82a04, user_id=0cc924e2dc8843c2873f0aba5e7ef5c5, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:15.051161 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727ac4af-86d0-4115-9c86-07eb2417e78e admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:15.063978 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1426/2866] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:31:15 2022] PUT /identity/v3/projects/168b55fa58274864a658fda1fdc82a04/users/0cc924e2dc8843c2873f0aba5e7ef5c5/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:31:15.074197 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcbb83d-1403-4cbe-aa11-6840cdfa364a None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:15.074481 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcbb83d-1403-4cbe-aa11-6840cdfa364a None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:15.074757 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcbb83d-1403-4cbe-aa11-6840cdfa364a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:15.102763 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-afcbb83d-1403-4cbe-aa11-6840cdfa364a None None] MFA Rules not processed for user `0cc924e2dc8843c2873f0aba5e7ef5c5`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:31:15.104007 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-afcbb83d-1403-4cbe-aa11-6840cdfa364a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:15.168791 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1441/2867] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:31:15 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:31:15.173291 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727ac4af-86d0-4115-9c86-07eb2417e78e admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:15.236554 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727ac4af-86d0-4115-9c86-07eb2417e78e admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:15.241430 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6dcd741-2fd2-4f18-a68b-c1638e470059 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:15.242027 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dcd741-2fd2-4f18-a68b-c1638e470059 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:15.242306 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dcd741-2fd2-4f18-a68b-c1638e470059 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:15.242579 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dcd741-2fd2-4f18-a68b-c1638e470059 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:15.243201 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6dcd741-2fd2-4f18-a68b-c1638e470059 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:15.245120 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6dcd741-2fd2-4f18-a68b-c1638e470059 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:15.271471 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1427/2868] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:31:15 2022] POST /identity/v3/projects => generated 366 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:31:15.275719 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afcbb83d-1403-4cbe-aa11-6840cdfa364a None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:15.287082 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afcbb83d-1403-4cbe-aa11-6840cdfa364a None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:15.344376 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ccf0e6-0f1b-4b11-9403-90538dc403df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:15.344910 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ccf0e6-0f1b-4b11-9403-90538dc403df admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:15.345191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ccf0e6-0f1b-4b11-9403-90538dc403df admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:15.345470 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ccf0e6-0f1b-4b11-9403-90538dc403df admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:15.346103 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ccf0e6-0f1b-4b11-9403-90538dc403df admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:15.348655 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ccf0e6-0f1b-4b11-9403-90538dc403df admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:15.369476 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1442/2869] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:31:15 2022] POST /identity/v3/users => generated 290 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 16:31:15.373623 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6dcd741-2fd2-4f18-a68b-c1638e470059 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:15.383739 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6dcd741-2fd2-4f18-a68b-c1638e470059 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:15.388276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac06722-d949-4022-b383-fd9fe2be7ab0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:15.388862 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac06722-d949-4022-b383-fd9fe2be7ab0 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:15.389140 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac06722-d949-4022-b383-fd9fe2be7ab0 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:15.389412 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac06722-d949-4022-b383-fd9fe2be7ab0 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:15.390043 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac06722-d949-4022-b383-fd9fe2be7ab0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:15.391659 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac06722-d949-4022-b383-fd9fe2be7ab0 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:15.402938 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1428/2870] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:31:15 2022] GET /identity/v3/roles => generated 1427 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:31:15.407016 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ccf0e6-0f1b-4b11-9403-90538dc403df admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:15.417562 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ccf0e6-0f1b-4b11-9403-90538dc403df admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:15.422159 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f5d3fc-fe7c-4d70-a165-315f72d7bfe8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:15.422802 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f5d3fc-fe7c-4d70-a165-315f72d7bfe8 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:15.423080 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f5d3fc-fe7c-4d70-a165-315f72d7bfe8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:15.423353 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f5d3fc-fe7c-4d70-a165-315f72d7bfe8 admin admin] PATH_INFO: `/v3/projects/273566072e8a4642946461bd61b0631f/users/24fd4d68ef02458eb138214074986a43/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:15.437950 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f5d3fc-fe7c-4d70-a165-315f72d7bfe8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=273566072e8a4642946461bd61b0631f, user_id=24fd4d68ef02458eb138214074986a43, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:15.440776 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f5d3fc-fe7c-4d70-a165-315f72d7bfe8 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:15.453368 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1443/2871] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:31:15 2022] PUT /identity/v3/projects/273566072e8a4642946461bd61b0631f/users/24fd4d68ef02458eb138214074986a43/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:31:15.457869 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac06722-d949-4022-b383-fd9fe2be7ab0 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:15.468459 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac06722-d949-4022-b383-fd9fe2be7ab0 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:15.489580 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:31:15.529976 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb7d2e1a-12d0-4204-b723-adaa5ca784de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:15.530603 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7d2e1a-12d0-4204-b723-adaa5ca784de admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:15.530866 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7d2e1a-12d0-4204-b723-adaa5ca784de admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:15.531137 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7d2e1a-12d0-4204-b723-adaa5ca784de admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:15.531818 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb7d2e1a-12d0-4204-b723-adaa5ca784de admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:15.533417 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb7d2e1a-12d0-4204-b723-adaa5ca784de admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:15.545117 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1429/2872] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:31:15 2022] GET /identity/v3/roles => generated 1427 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 16:31:15.549408 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f5d3fc-fe7c-4d70-a165-315f72d7bfe8 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:15.555436 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:31:15.562190 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f5d3fc-fe7c-4d70-a165-315f72d7bfe8 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:15.567258 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53e23bf-d9f7-4be3-a186-5891ede0bd9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:15.567957 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53e23bf-d9f7-4be3-a186-5891ede0bd9a admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:15.568242 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53e23bf-d9f7-4be3-a186-5891ede0bd9a admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:15.568520 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53e23bf-d9f7-4be3-a186-5891ede0bd9a admin admin] PATH_INFO: `/v3/projects/273566072e8a4642946461bd61b0631f/users/24fd4d68ef02458eb138214074986a43/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:15.570449 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c53e23bf-d9f7-4be3-a186-5891ede0bd9a admin admin] RBAC: Authorizing `identity:create_grant(project_id=273566072e8a4642946461bd61b0631f, user_id=24fd4d68ef02458eb138214074986a43, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:15.573141 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c53e23bf-d9f7-4be3-a186-5891ede0bd9a admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:15.583979 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1444/2873] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:31:15 2022] PUT /identity/v3/projects/273566072e8a4642946461bd61b0631f/users/24fd4d68ef02458eb138214074986a43/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:31:15.595441 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efa469a-9bae-4e80-ac4f-cf0907b3cb5f None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:15.595693 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efa469a-9bae-4e80-ac4f-cf0907b3cb5f None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:15.595970 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efa469a-9bae-4e80-ac4f-cf0907b3cb5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:15.623802 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-1efa469a-9bae-4e80-ac4f-cf0907b3cb5f None None] MFA Rules not processed for user `24fd4d68ef02458eb138214074986a43`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:31:15.625007 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-1efa469a-9bae-4e80-ac4f-cf0907b3cb5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:15.692862 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1430/2874] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:31:15 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:31:15.703853 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53e23bf-d9f7-4be3-a186-5891ede0bd9a admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:15.705711 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-c53e23bf-d9f7-4be3-a186-5891ede0bd9a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:15.765561 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53e23bf-d9f7-4be3-a186-5891ede0bd9a admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:15.769737 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901f1032-b95c-488e-bb27-747003847305 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:15.770344 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901f1032-b95c-488e-bb27-747003847305 service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:15.770627 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901f1032-b95c-488e-bb27-747003847305 service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:15.770903 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901f1032-b95c-488e-bb27-747003847305 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:15.780339 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901f1032-b95c-488e-bb27-747003847305 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:15.781782 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901f1032-b95c-488e-bb27-747003847305 service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:15.791953 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1445/2875] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:31:15 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:31:16.213473 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efa469a-9bae-4e80-ac4f-cf0907b3cb5f None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:16.278480 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efa469a-9bae-4e80-ac4f-cf0907b3cb5f None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:16.282998 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d479c765-42d7-4b1e-8c8c-3eb3451389b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:16.283715 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d479c765-42d7-4b1e-8c8c-3eb3451389b5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:16.284033 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d479c765-42d7-4b1e-8c8c-3eb3451389b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:16.285427 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d479c765-42d7-4b1e-8c8c-3eb3451389b5 admin admin] PATH_INFO: `/v3/users/0cc924e2dc8843c2873f0aba5e7ef5c5` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:16.286577 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d479c765-42d7-4b1e-8c8c-3eb3451389b5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0cc924e2dc8843c2873f0aba5e7ef5c5)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:16.288534 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d479c765-42d7-4b1e-8c8c-3eb3451389b5 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:16.336929 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d479c765-42d7-4b1e-8c8c-3eb3451389b5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0cc924e2dc8843c2873f0aba5e7ef5c5'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:16.337632 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d479c765-42d7-4b1e-8c8c-3eb3451389b5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0cc924e2dc8843c2873f0aba5e7ef5c5'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:16.338880 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:31:16.346248 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d479c765-42d7-4b1e-8c8c-3eb3451389b5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0cc924e2dc8843c2873f0aba5e7ef5c5'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:16.359124 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d479c765-42d7-4b1e-8c8c-3eb3451389b5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0cc924e2dc8843c2873f0aba5e7ef5c5'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:16.359752 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d479c765-42d7-4b1e-8c8c-3eb3451389b5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0cc924e2dc8843c2873f0aba5e7ef5c5'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:16.372684 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1431/2876] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:31:16 2022] DELETE /identity/v3/users/0cc924e2dc8843c2873f0aba5e7ef5c5 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:31:16.377192 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055176ad-4d3d-4ccd-891e-c0cae7e96e60 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:16.377480 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055176ad-4d3d-4ccd-891e-c0cae7e96e60 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:16.378249 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055176ad-4d3d-4ccd-891e-c0cae7e96e60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:16.397282 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-055176ad-4d3d-4ccd-891e-c0cae7e96e60 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:31:16.398471 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-055176ad-4d3d-4ccd-891e-c0cae7e96e60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:16.457495 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1446/2877] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:31:16 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:31:16.467354 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d479c765-42d7-4b1e-8c8c-3eb3451389b5 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:16.469013 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-d479c765-42d7-4b1e-8c8c-3eb3451389b5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:16.527024 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d479c765-42d7-4b1e-8c8c-3eb3451389b5 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:16.530675 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba6766ab-8617-4b99-8b0d-af6c1e69c966 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:16.531276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6766ab-8617-4b99-8b0d-af6c1e69c966 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:16.531560 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6766ab-8617-4b99-8b0d-af6c1e69c966 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:16.531864 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6766ab-8617-4b99-8b0d-af6c1e69c966 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:16.542365 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba6766ab-8617-4b99-8b0d-af6c1e69c966 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:16.543838 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba6766ab-8617-4b99-8b0d-af6c1e69c966 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:16.555053 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1432/2878] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:31:16 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 16:31:16.830665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055176ad-4d3d-4ccd-891e-c0cae7e96e60 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:16.833460 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-055176ad-4d3d-4ccd-891e-c0cae7e96e60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:16.851666 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055176ad-4d3d-4ccd-891e-c0cae7e96e60 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:16.856320 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ab2887-259c-4757-884b-5d3068d364e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:16.856964 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ab2887-259c-4757-884b-5d3068d364e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:16.857246 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ab2887-259c-4757-884b-5d3068d364e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:16.857522 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ab2887-259c-4757-884b-5d3068d364e2 admin admin] PATH_INFO: `/v3/projects/168b55fa58274864a658fda1fdc82a04` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:16.858675 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ab2887-259c-4757-884b-5d3068d364e2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=168b55fa58274864a658fda1fdc82a04)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:16.860847 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ab2887-259c-4757-884b-5d3068d364e2 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:16.909961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-00ab2887-259c-4757-884b-5d3068d364e2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '168b55fa58274864a658fda1fdc82a04', 'request_id': 'req-00ab2887-259c-4757-884b-5d3068d364e2'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:16.916266 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-00ab2887-259c-4757-884b-5d3068d364e2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '168b55fa58274864a658fda1fdc82a04', 'request_id': 'req-00ab2887-259c-4757-884b-5d3068d364e2'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:16.922456 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-00ab2887-259c-4757-884b-5d3068d364e2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '168b55fa58274864a658fda1fdc82a04', 'request_id': 'req-00ab2887-259c-4757-884b-5d3068d364e2'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:16.936308 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-00ab2887-259c-4757-884b-5d3068d364e2 admin admin] The token cache is being invalidate because project 168b55fa58274864a658fda1fdc82a04 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:31:16.936603 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-00ab2887-259c-4757-884b-5d3068d364e2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:16.937254 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-00ab2887-259c-4757-884b-5d3068d364e2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:16.938741 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1447/2879] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:31:16 2022] DELETE /identity/v3/projects/168b55fa58274864a658fda1fdc82a04 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:31:16.943578 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba6766ab-8617-4b99-8b0d-af6c1e69c966 service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:16.953246 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba6766ab-8617-4b99-8b0d-af6c1e69c966 service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:17.011281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40809d7b-a408-48c8-8ccd-417302ea744a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:17.011934 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40809d7b-a408-48c8-8ccd-417302ea744a admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:17.012215 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40809d7b-a408-48c8-8ccd-417302ea744a admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:17.012493 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40809d7b-a408-48c8-8ccd-417302ea744a admin admin] PATH_INFO: `/v3/users/24fd4d68ef02458eb138214074986a43` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:17.013625 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40809d7b-a408-48c8-8ccd-417302ea744a admin admin] RBAC: Authorizing `identity:delete_user(user_id=24fd4d68ef02458eb138214074986a43)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:17.015535 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40809d7b-a408-48c8-8ccd-417302ea744a admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:17.062344 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-40809d7b-a408-48c8-8ccd-417302ea744a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '24fd4d68ef02458eb138214074986a43'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:17.063077 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-40809d7b-a408-48c8-8ccd-417302ea744a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '24fd4d68ef02458eb138214074986a43'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:17.070751 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-40809d7b-a408-48c8-8ccd-417302ea744a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '24fd4d68ef02458eb138214074986a43'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:17.080739 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:31:17.083281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-40809d7b-a408-48c8-8ccd-417302ea744a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '24fd4d68ef02458eb138214074986a43'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:17.084213 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-40809d7b-a408-48c8-8ccd-417302ea744a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '24fd4d68ef02458eb138214074986a43'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:17.098191 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1433/2880] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:31:16 2022] DELETE /identity/v3/users/24fd4d68ef02458eb138214074986a43 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:31:17.624328 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ab2887-259c-4757-884b-5d3068d364e2 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:17.625856 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-00ab2887-259c-4757-884b-5d3068d364e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:17.642605 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ab2887-259c-4757-884b-5d3068d364e2 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:17.657085 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:31:17.704909 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdfc636d-71fd-4914-8944-7b03fb43af02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:17.705544 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfc636d-71fd-4914-8944-7b03fb43af02 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:17.706329 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfc636d-71fd-4914-8944-7b03fb43af02 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:17.706663 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfc636d-71fd-4914-8944-7b03fb43af02 admin admin] PATH_INFO: `/v3/projects/273566072e8a4642946461bd61b0631f` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:17.707857 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdfc636d-71fd-4914-8944-7b03fb43af02 admin admin] RBAC: Authorizing `identity:delete_project(project_id=273566072e8a4642946461bd61b0631f)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:17.709764 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdfc636d-71fd-4914-8944-7b03fb43af02 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:17.757444 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-fdfc636d-71fd-4914-8944-7b03fb43af02 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '273566072e8a4642946461bd61b0631f', 'request_id': 'req-fdfc636d-71fd-4914-8944-7b03fb43af02'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:17.763415 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-fdfc636d-71fd-4914-8944-7b03fb43af02 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '273566072e8a4642946461bd61b0631f', 'request_id': 'req-fdfc636d-71fd-4914-8944-7b03fb43af02'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:17.769531 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-fdfc636d-71fd-4914-8944-7b03fb43af02 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '273566072e8a4642946461bd61b0631f', 'request_id': 'req-fdfc636d-71fd-4914-8944-7b03fb43af02'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:17.781981 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-fdfc636d-71fd-4914-8944-7b03fb43af02 admin admin] The token cache is being invalidate because project 273566072e8a4642946461bd61b0631f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:31:17.782270 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-fdfc636d-71fd-4914-8944-7b03fb43af02 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:17.782922 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-fdfc636d-71fd-4914-8944-7b03fb43af02 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:17.785125 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1448/2881] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:31:17 2022] DELETE /identity/v3/projects/273566072e8a4642946461bd61b0631f => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:31:17.790147 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acfcb0f-f9da-4765-8d84-d909b30caf83 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:17.790648 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acfcb0f-f9da-4765-8d84-d909b30caf83 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:17.790740 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acfcb0f-f9da-4765-8d84-d909b30caf83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:17.810193 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-6acfcb0f-f9da-4765-8d84-d909b30caf83 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:31:17.811393 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-6acfcb0f-f9da-4765-8d84-d909b30caf83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:17.880637 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1434/2882] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:31:17 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:31:17.888963 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66b4b40-b63e-45f9-97aa-b885e7b0ed84 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:17.889448 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66b4b40-b63e-45f9-97aa-b885e7b0ed84 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:17.889562 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66b4b40-b63e-45f9-97aa-b885e7b0ed84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:17.911349 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-d66b4b40-b63e-45f9-97aa-b885e7b0ed84 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:31:17.912614 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-d66b4b40-b63e-45f9-97aa-b885e7b0ed84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:17.920764 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1449/2883] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:31:17 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:31:17.925544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6acfcb0f-f9da-4765-8d84-d909b30caf83 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:17.935413 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6acfcb0f-f9da-4765-8d84-d909b30caf83 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:17.939998 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c467884-c4ba-49dd-aacf-14a7bf95bca2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:17.940704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c467884-c4ba-49dd-aacf-14a7bf95bca2 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:17.940984 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c467884-c4ba-49dd-aacf-14a7bf95bca2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:17.941392 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c467884-c4ba-49dd-aacf-14a7bf95bca2 admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:17.941908 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c467884-c4ba-49dd-aacf-14a7bf95bca2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 16:31:17.942215 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c467884-c4ba-49dd-aacf-14a7bf95bca2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:17.943836 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c467884-c4ba-49dd-aacf-14a7bf95bca2 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:17.955171 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1435/2884] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 16:31:17 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 16:31:17.959463 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d66b4b40-b63e-45f9-97aa-b885e7b0ed84 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:17.968926 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d66b4b40-b63e-45f9-97aa-b885e7b0ed84 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:17.974544 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99687d5c-d741-42f7-a2bd-0b1668247f69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:17.975134 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99687d5c-d741-42f7-a2bd-0b1668247f69 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:17.975410 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99687d5c-d741-42f7-a2bd-0b1668247f69 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:17.975717 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99687d5c-d741-42f7-a2bd-0b1668247f69 admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:17.976354 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99687d5c-d741-42f7-a2bd-0b1668247f69 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:17.978249 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99687d5c-d741-42f7-a2bd-0b1668247f69 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:18.006195 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1450/2885] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:31:17 2022] POST /identity/v3/projects => generated 402 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:31:18.010297 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c467884-c4ba-49dd-aacf-14a7bf95bca2 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:18.020253 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c467884-c4ba-49dd-aacf-14a7bf95bca2 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:18.076277 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb6a4d1-1f8d-4bdb-9fcd-dfd230660ed6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:18.076872 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb6a4d1-1f8d-4bdb-9fcd-dfd230660ed6 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:18.077149 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb6a4d1-1f8d-4bdb-9fcd-dfd230660ed6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:18.077419 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb6a4d1-1f8d-4bdb-9fcd-dfd230660ed6 admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:18.078037 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb6a4d1-1f8d-4bdb-9fcd-dfd230660ed6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:18.079997 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb6a4d1-1f8d-4bdb-9fcd-dfd230660ed6 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:18.101792 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1436/2886] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:31:18 2022] POST /identity/v3/users => generated 302 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:31:18.107150 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99687d5c-d741-42f7-a2bd-0b1668247f69 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:18.120965 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99687d5c-d741-42f7-a2bd-0b1668247f69 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:18.125747 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c4941e-6dd0-479a-b8bb-5d0817df4af7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:18.126322 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c4941e-6dd0-479a-b8bb-5d0817df4af7 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:18.126609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c4941e-6dd0-479a-b8bb-5d0817df4af7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:18.126886 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c4941e-6dd0-479a-b8bb-5d0817df4af7 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:18.127535 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c4941e-6dd0-479a-b8bb-5d0817df4af7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:18.129766 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c4941e-6dd0-479a-b8bb-5d0817df4af7 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:18.140417 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1451/2887] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:31:18 2022] GET /identity/v3/roles => generated 1427 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:31:18.144486 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb6a4d1-1f8d-4bdb-9fcd-dfd230660ed6 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:18.154867 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb6a4d1-1f8d-4bdb-9fcd-dfd230660ed6 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:18.160711 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47498531-5f2b-4c15-be80-71d0649c499a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:18.161310 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47498531-5f2b-4c15-be80-71d0649c499a admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:18.161593 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47498531-5f2b-4c15-be80-71d0649c499a admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:18.161865 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47498531-5f2b-4c15-be80-71d0649c499a admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:18.162514 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47498531-5f2b-4c15-be80-71d0649c499a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:18.164151 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47498531-5f2b-4c15-be80-71d0649c499a admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:18.175690 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1437/2888] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:31:18 2022] GET /identity/v3/roles => generated 1427 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:31:18.179838 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c4941e-6dd0-479a-b8bb-5d0817df4af7 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:18.189924 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c4941e-6dd0-479a-b8bb-5d0817df4af7 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:18.194910 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a051482-e94f-4ab9-84db-df37d72d736d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:18.195586 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a051482-e94f-4ab9-84db-df37d72d736d admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:18.195901 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a051482-e94f-4ab9-84db-df37d72d736d admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:18.196175 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a051482-e94f-4ab9-84db-df37d72d736d admin admin] PATH_INFO: `/v3/projects/c4c67e809d9f4f4b97a8c6bf9914850c/users/72af5f63e7a042e094c76f9b5644f9e2/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:18.213309 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a051482-e94f-4ab9-84db-df37d72d736d admin admin] RBAC: Authorizing `identity:create_grant(project_id=c4c67e809d9f4f4b97a8c6bf9914850c, user_id=72af5f63e7a042e094c76f9b5644f9e2, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:18.216038 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a051482-e94f-4ab9-84db-df37d72d736d admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:18.230423 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1452/2889] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:31:18 2022] PUT /identity/v3/projects/c4c67e809d9f4f4b97a8c6bf9914850c/users/72af5f63e7a042e094c76f9b5644f9e2/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:31:18.241511 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e428bb20-2f26-4ae0-831e-1b287732f625 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:18.241806 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e428bb20-2f26-4ae0-831e-1b287732f625 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:18.242074 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e428bb20-2f26-4ae0-831e-1b287732f625 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:18.270454 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-e428bb20-2f26-4ae0-831e-1b287732f625 None None] MFA Rules not processed for user `72af5f63e7a042e094c76f9b5644f9e2`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:31:18.271702 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-e428bb20-2f26-4ae0-831e-1b287732f625 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:18.313284 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:DELETE=2,keystone:SELECT=94,keystone:INSERT=4 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:31:18.337960 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1438/2890] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:31:18 2022] POST /identity/v3/auth/tokens => generated 3288 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:31:18.342533 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a051482-e94f-4ab9-84db-df37d72d736d admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:18.387579 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:31:18.408214 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a051482-e94f-4ab9-84db-df37d72d736d admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:18.414319 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb7dbcc-088b-4e6c-b19e-e085db8afbab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:18.414914 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb7dbcc-088b-4e6c-b19e-e085db8afbab admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:18.415191 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb7dbcc-088b-4e6c-b19e-e085db8afbab admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:18.415461 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb7dbcc-088b-4e6c-b19e-e085db8afbab admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:18.416109 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb7dbcc-088b-4e6c-b19e-e085db8afbab admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:18.418000 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb7dbcc-088b-4e6c-b19e-e085db8afbab admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:18.445373 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1453/2891] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:31:18 2022] POST /identity/v3/projects => generated 402 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:31:18.449537 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e428bb20-2f26-4ae0-831e-1b287732f625 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:18.460263 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e428bb20-2f26-4ae0-831e-1b287732f625 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:18.516041 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0591abf7-10ec-447f-ae81-5f00c614d93b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:18.516634 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0591abf7-10ec-447f-ae81-5f00c614d93b admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:18.516915 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0591abf7-10ec-447f-ae81-5f00c614d93b admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:18.517189 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0591abf7-10ec-447f-ae81-5f00c614d93b admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:18.517823 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0591abf7-10ec-447f-ae81-5f00c614d93b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:18.519782 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0591abf7-10ec-447f-ae81-5f00c614d93b admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:18.539266 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1439/2892] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:31:18 2022] POST /identity/v3/users => generated 308 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:31:18.543435 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb7dbcc-088b-4e6c-b19e-e085db8afbab admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:18.554813 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb7dbcc-088b-4e6c-b19e-e085db8afbab admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:18.559521 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d7551f-ba8d-4de0-926b-29be548fca70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:18.560198 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d7551f-ba8d-4de0-926b-29be548fca70 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:18.560474 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d7551f-ba8d-4de0-926b-29be548fca70 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:18.560744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d7551f-ba8d-4de0-926b-29be548fca70 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:18.561386 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d7551f-ba8d-4de0-926b-29be548fca70 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:18.562968 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d7551f-ba8d-4de0-926b-29be548fca70 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:18.574907 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1454/2893] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:31:18 2022] GET /identity/v3/roles => generated 1427 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:31:18.579209 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0591abf7-10ec-447f-ae81-5f00c614d93b admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:18.589818 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0591abf7-10ec-447f-ae81-5f00c614d93b admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:18.594642 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a402d131-32ea-4e1e-8414-875a9e4688f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:18.595280 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a402d131-32ea-4e1e-8414-875a9e4688f0 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:18.595558 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a402d131-32ea-4e1e-8414-875a9e4688f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:18.595856 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a402d131-32ea-4e1e-8414-875a9e4688f0 admin admin] PATH_INFO: `/v3/projects/00d6308322e04ca28cd1802d6962e267/users/baee841e974446819ccb2aa9d2e89edc/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:18.611841 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a402d131-32ea-4e1e-8414-875a9e4688f0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=00d6308322e04ca28cd1802d6962e267, user_id=baee841e974446819ccb2aa9d2e89edc, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:18.616684 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a402d131-32ea-4e1e-8414-875a9e4688f0 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:18.631843 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1440/2894] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:31:18 2022] PUT /identity/v3/projects/00d6308322e04ca28cd1802d6962e267/users/baee841e974446819ccb2aa9d2e89edc/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:31:18.638312 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d7551f-ba8d-4de0-926b-29be548fca70 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:18.649166 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d7551f-ba8d-4de0-926b-29be548fca70 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:18.705785 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae37e334-3718-4b42-8b1c-262bdeed2a2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:18.706830 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae37e334-3718-4b42-8b1c-262bdeed2a2f admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:18.707114 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae37e334-3718-4b42-8b1c-262bdeed2a2f admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:18.707391 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae37e334-3718-4b42-8b1c-262bdeed2a2f admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:18.708082 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae37e334-3718-4b42-8b1c-262bdeed2a2f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:18.709692 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae37e334-3718-4b42-8b1c-262bdeed2a2f admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:18.720273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1455/2895] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:31:18 2022] GET /identity/v3/roles => generated 1427 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:31:18.725666 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a402d131-32ea-4e1e-8414-875a9e4688f0 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:18.736406 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a402d131-32ea-4e1e-8414-875a9e4688f0 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:18.741794 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7448af8a-3a90-43f4-9aa6-6beb7b8481e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:18.742441 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7448af8a-3a90-43f4-9aa6-6beb7b8481e7 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:18.742718 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7448af8a-3a90-43f4-9aa6-6beb7b8481e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:18.742985 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7448af8a-3a90-43f4-9aa6-6beb7b8481e7 admin admin] PATH_INFO: `/v3/projects/00d6308322e04ca28cd1802d6962e267/users/baee841e974446819ccb2aa9d2e89edc/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:18.744895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7448af8a-3a90-43f4-9aa6-6beb7b8481e7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=00d6308322e04ca28cd1802d6962e267, user_id=baee841e974446819ccb2aa9d2e89edc, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:18.747526 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7448af8a-3a90-43f4-9aa6-6beb7b8481e7 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:18.758140 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1441/2896] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:31:18 2022] PUT /identity/v3/projects/00d6308322e04ca28cd1802d6962e267/users/baee841e974446819ccb2aa9d2e89edc/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:31:18.769867 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbd285d-1cd1-4073-a8e6-17753248e970 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:18.770165 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbd285d-1cd1-4073-a8e6-17753248e970 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:18.770444 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbd285d-1cd1-4073-a8e6-17753248e970 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:18.798500 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-5fbd285d-1cd1-4073-a8e6-17753248e970 None None] MFA Rules not processed for user `baee841e974446819ccb2aa9d2e89edc`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:31:18.800439 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-5fbd285d-1cd1-4073-a8e6-17753248e970 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:18.871476 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1456/2897] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:31:18 2022] POST /identity/v3/auth/tokens => generated 3355 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:31:18.882346 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7448af8a-3a90-43f4-9aa6-6beb7b8481e7 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:18.884127 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-7448af8a-3a90-43f4-9aa6-6beb7b8481e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:18.944278 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7448af8a-3a90-43f4-9aa6-6beb7b8481e7 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:18.948007 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a91b1bf-e083-4b4e-9cb3-b9d96f3bdb87 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:18.948614 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a91b1bf-e083-4b4e-9cb3-b9d96f3bdb87 service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:18.948895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a91b1bf-e083-4b4e-9cb3-b9d96f3bdb87 service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:18.949169 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a91b1bf-e083-4b4e-9cb3-b9d96f3bdb87 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:18.958930 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a91b1bf-e083-4b4e-9cb3-b9d96f3bdb87 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:18.960371 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a91b1bf-e083-4b4e-9cb3-b9d96f3bdb87 service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:18.969995 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1442/2898] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:31:18 2022] GET /identity/v3/auth/tokens => generated 3355 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 16:31:19.046721 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbd285d-1cd1-4073-a8e6-17753248e970 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:19.056813 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbd285d-1cd1-4073-a8e6-17753248e970 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:19.060546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35eaf36a-0236-48de-989e-0909943caa4e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:19.061134 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35eaf36a-0236-48de-989e-0909943caa4e service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:19.061414 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35eaf36a-0236-48de-989e-0909943caa4e service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:19.061687 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35eaf36a-0236-48de-989e-0909943caa4e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:19.095617 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:31:19.120450 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35eaf36a-0236-48de-989e-0909943caa4e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:19.121891 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35eaf36a-0236-48de-989e-0909943caa4e service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:19.130851 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1457/2899] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:31:19 2022] GET /identity/v3/auth/tokens => generated 3288 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:31:23.371545 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a91b1bf-e083-4b4e-9cb3-b9d96f3bdb87 service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:23.412860 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:31:23.434653 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a91b1bf-e083-4b4e-9cb3-b9d96f3bdb87 service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:23.439105 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee70469-40c0-4620-aff1-e91e82789cb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:23.439744 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee70469-40c0-4620-aff1-e91e82789cb7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:23.440022 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee70469-40c0-4620-aff1-e91e82789cb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:23.440297 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee70469-40c0-4620-aff1-e91e82789cb7 admin admin] PATH_INFO: `/v3/users/72af5f63e7a042e094c76f9b5644f9e2` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:23.441418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee70469-40c0-4620-aff1-e91e82789cb7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=72af5f63e7a042e094c76f9b5644f9e2)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:23.444048 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee70469-40c0-4620-aff1-e91e82789cb7 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:23.490538 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-eee70469-40c0-4620-aff1-e91e82789cb7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '72af5f63e7a042e094c76f9b5644f9e2'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:23.491287 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-eee70469-40c0-4620-aff1-e91e82789cb7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '72af5f63e7a042e094c76f9b5644f9e2'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:23.498221 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-eee70469-40c0-4620-aff1-e91e82789cb7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '72af5f63e7a042e094c76f9b5644f9e2'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:23.510473 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-eee70469-40c0-4620-aff1-e91e82789cb7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '72af5f63e7a042e094c76f9b5644f9e2'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:23.511127 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-eee70469-40c0-4620-aff1-e91e82789cb7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '72af5f63e7a042e094c76f9b5644f9e2'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:23.523672 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1443/2900] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:31:23 2022] DELETE /identity/v3/users/72af5f63e7a042e094c76f9b5644f9e2 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:31:23.528146 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f360c1b-9889-4401-b65c-9bf657001e44 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:23.528609 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f360c1b-9889-4401-b65c-9bf657001e44 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:23.528721 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f360c1b-9889-4401-b65c-9bf657001e44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:23.547708 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-4f360c1b-9889-4401-b65c-9bf657001e44 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:31:23.548916 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-4f360c1b-9889-4401-b65c-9bf657001e44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:23.608569 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1458/2901] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:31:23 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:31:23.619114 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee70469-40c0-4620-aff1-e91e82789cb7 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:23.620589 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-eee70469-40c0-4620-aff1-e91e82789cb7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:23.678500 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee70469-40c0-4620-aff1-e91e82789cb7 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:23.682190 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dbe598a-8c66-4d20-84a2-f4ffc9825a86 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:23.682787 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbe598a-8c66-4d20-84a2-f4ffc9825a86 service neutron] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:23.683068 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbe598a-8c66-4d20-84a2-f4ffc9825a86 service neutron] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:23.683341 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbe598a-8c66-4d20-84a2-f4ffc9825a86 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:23.693518 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dbe598a-8c66-4d20-84a2-f4ffc9825a86 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:23.694998 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dbe598a-8c66-4d20-84a2-f4ffc9825a86 service neutron] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:23.704836 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1444/2902] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:31:23 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:31:23.993123 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f360c1b-9889-4401-b65c-9bf657001e44 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:23.994813 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-4f360c1b-9889-4401-b65c-9bf657001e44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:24.008611 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f360c1b-9889-4401-b65c-9bf657001e44 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:24.013218 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511f5a11-4248-4135-9cde-4340c77b1e03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:24.013850 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511f5a11-4248-4135-9cde-4340c77b1e03 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:24.014129 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511f5a11-4248-4135-9cde-4340c77b1e03 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:24.014400 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511f5a11-4248-4135-9cde-4340c77b1e03 admin admin] PATH_INFO: `/v3/projects/c4c67e809d9f4f4b97a8c6bf9914850c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:24.015516 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511f5a11-4248-4135-9cde-4340c77b1e03 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c4c67e809d9f4f4b97a8c6bf9914850c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:24.017429 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511f5a11-4248-4135-9cde-4340c77b1e03 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:24.064094 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-511f5a11-4248-4135-9cde-4340c77b1e03 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c4c67e809d9f4f4b97a8c6bf9914850c', 'request_id': 'req-511f5a11-4248-4135-9cde-4340c77b1e03'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:24.070851 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-511f5a11-4248-4135-9cde-4340c77b1e03 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c4c67e809d9f4f4b97a8c6bf9914850c', 'request_id': 'req-511f5a11-4248-4135-9cde-4340c77b1e03'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:24.076541 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-511f5a11-4248-4135-9cde-4340c77b1e03 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c4c67e809d9f4f4b97a8c6bf9914850c', 'request_id': 'req-511f5a11-4248-4135-9cde-4340c77b1e03'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:24.089500 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-511f5a11-4248-4135-9cde-4340c77b1e03 admin admin] The token cache is being invalidate because project c4c67e809d9f4f4b97a8c6bf9914850c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:31:24.089792 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-511f5a11-4248-4135-9cde-4340c77b1e03 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:24.090409 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-511f5a11-4248-4135-9cde-4340c77b1e03 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:24.091931 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1459/2903] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:31:23 2022] DELETE /identity/v3/projects/c4c67e809d9f4f4b97a8c6bf9914850c => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:31:24.096245 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dbe598a-8c66-4d20-84a2-f4ffc9825a86 service neutron] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:24.106096 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dbe598a-8c66-4d20-84a2-f4ffc9825a86 service neutron] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:24.162331 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81667e33-2b62-459e-84b0-a49d61c926de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:24.162967 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81667e33-2b62-459e-84b0-a49d61c926de admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:24.163253 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81667e33-2b62-459e-84b0-a49d61c926de admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:24.163532 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81667e33-2b62-459e-84b0-a49d61c926de admin admin] PATH_INFO: `/v3/users/baee841e974446819ccb2aa9d2e89edc` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:24.164733 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81667e33-2b62-459e-84b0-a49d61c926de admin admin] RBAC: Authorizing `identity:delete_user(user_id=baee841e974446819ccb2aa9d2e89edc)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:24.166700 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81667e33-2b62-459e-84b0-a49d61c926de admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:24.168214 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:31:24.215016 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-81667e33-2b62-459e-84b0-a49d61c926de admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'baee841e974446819ccb2aa9d2e89edc'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:24.215681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-81667e33-2b62-459e-84b0-a49d61c926de admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'baee841e974446819ccb2aa9d2e89edc'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:24.222685 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-81667e33-2b62-459e-84b0-a49d61c926de admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'baee841e974446819ccb2aa9d2e89edc'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:24.235457 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-81667e33-2b62-459e-84b0-a49d61c926de admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'baee841e974446819ccb2aa9d2e89edc'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:24.236161 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-81667e33-2b62-459e-84b0-a49d61c926de admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'baee841e974446819ccb2aa9d2e89edc'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:24.250028 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1445/2904] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:31:24 2022] DELETE /identity/v3/users/baee841e974446819ccb2aa9d2e89edc => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:31:24.527724 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511f5a11-4248-4135-9cde-4340c77b1e03 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:24.529214 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-511f5a11-4248-4135-9cde-4340c77b1e03 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:24.543103 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511f5a11-4248-4135-9cde-4340c77b1e03 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:24.605647 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7a1216-17f0-418e-ab02-6561e69afa58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:24.606202 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7a1216-17f0-418e-ab02-6561e69afa58 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:24.606486 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7a1216-17f0-418e-ab02-6561e69afa58 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:24.606764 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7a1216-17f0-418e-ab02-6561e69afa58 admin admin] PATH_INFO: `/v3/projects/00d6308322e04ca28cd1802d6962e267` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:24.607942 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da7a1216-17f0-418e-ab02-6561e69afa58 admin admin] RBAC: Authorizing `identity:delete_project(project_id=00d6308322e04ca28cd1802d6962e267)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:24.609848 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da7a1216-17f0-418e-ab02-6561e69afa58 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:24.611301 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:31:24.660002 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-da7a1216-17f0-418e-ab02-6561e69afa58 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '00d6308322e04ca28cd1802d6962e267', 'request_id': 'req-da7a1216-17f0-418e-ab02-6561e69afa58'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:24.666847 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-da7a1216-17f0-418e-ab02-6561e69afa58 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '00d6308322e04ca28cd1802d6962e267', 'request_id': 'req-da7a1216-17f0-418e-ab02-6561e69afa58'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:24.672341 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-da7a1216-17f0-418e-ab02-6561e69afa58 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '00d6308322e04ca28cd1802d6962e267', 'request_id': 'req-da7a1216-17f0-418e-ab02-6561e69afa58'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:24.687365 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-da7a1216-17f0-418e-ab02-6561e69afa58 admin admin] The token cache is being invalidate because project 00d6308322e04ca28cd1802d6962e267 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88217) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:31:24.687682 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-da7a1216-17f0-418e-ab02-6561e69afa58 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:24.688310 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-da7a1216-17f0-418e-ab02-6561e69afa58 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:24.689804 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1460/2905] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:31:24 2022] DELETE /identity/v3/projects/00d6308322e04ca28cd1802d6962e267 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:31:24.696024 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bac9c9-1b01-4aad-9509-b9eabfd7d47c None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:24.696534 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bac9c9-1b01-4aad-9509-b9eabfd7d47c None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:24.696629 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bac9c9-1b01-4aad-9509-b9eabfd7d47c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:24.716321 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-a0bac9c9-1b01-4aad-9509-b9eabfd7d47c None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:31:24.717591 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-a0bac9c9-1b01-4aad-9509-b9eabfd7d47c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:24.777004 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1446/2906] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:31:24 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:31:24.784956 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ccdf6f-c7d9-4f4a-8699-35e379c7d5fb None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:24.785768 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ccdf6f-c7d9-4f4a-8699-35e379c7d5fb None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:24.786009 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ccdf6f-c7d9-4f4a-8699-35e379c7d5fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:24.806106 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-78ccdf6f-c7d9-4f4a-8699-35e379c7d5fb None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:31:24.807310 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-78ccdf6f-c7d9-4f4a-8699-35e379c7d5fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:24.815163 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1461/2907] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:31:24 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:31:24.819525 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bac9c9-1b01-4aad-9509-b9eabfd7d47c None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:24.829294 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bac9c9-1b01-4aad-9509-b9eabfd7d47c None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:24.833963 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e98ba0-7f75-493d-bad3-e6022dbba8d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:24.834566 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e98ba0-7f75-493d-bad3-e6022dbba8d8 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:24.834875 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e98ba0-7f75-493d-bad3-e6022dbba8d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:24.835144 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e98ba0-7f75-493d-bad3-e6022dbba8d8 admin admin] PATH_INFO: `/v3/domains` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:24.835828 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e98ba0-7f75-493d-bad3-e6022dbba8d8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88218) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 16:31:24.836133 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e98ba0-7f75-493d-bad3-e6022dbba8d8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:24.837714 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e98ba0-7f75-493d-bad3-e6022dbba8d8 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:24.850462 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1447/2908] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 16:31:24 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 16:31:24.854966 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ccdf6f-c7d9-4f4a-8699-35e379c7d5fb None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:24.865306 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ccdf6f-c7d9-4f4a-8699-35e379c7d5fb None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:24.870294 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3970e55-0647-4745-a89a-b515e7da2687 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:24.870890 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3970e55-0647-4745-a89a-b515e7da2687 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:24.871171 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3970e55-0647-4745-a89a-b515e7da2687 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:24.871447 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3970e55-0647-4745-a89a-b515e7da2687 admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:24.872124 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3970e55-0647-4745-a89a-b515e7da2687 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:24.874026 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3970e55-0647-4745-a89a-b515e7da2687 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:24.902367 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1462/2909] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:31:24 2022] POST /identity/v3/projects => generated 384 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:31:24.906505 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e98ba0-7f75-493d-bad3-e6022dbba8d8 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:24.917722 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e98ba0-7f75-493d-bad3-e6022dbba8d8 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:24.980779 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161b23df-3bf8-4642-b862-e30e9948b0b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:24.981375 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161b23df-3bf8-4642-b862-e30e9948b0b2 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:24.981654 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161b23df-3bf8-4642-b862-e30e9948b0b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:24.981927 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161b23df-3bf8-4642-b862-e30e9948b0b2 admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:24.982547 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-161b23df-3bf8-4642-b862-e30e9948b0b2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:24.984516 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-161b23df-3bf8-4642-b862-e30e9948b0b2 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:24.995318 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:31:25.007895 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1448/2910] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:31:24 2022] POST /identity/v3/users => generated 293 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:31:25.011994 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3970e55-0647-4745-a89a-b515e7da2687 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:25.023139 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3970e55-0647-4745-a89a-b515e7da2687 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:25.027732 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779a8e02-4190-42ed-8de0-def7f9aadffa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:25.028330 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779a8e02-4190-42ed-8de0-def7f9aadffa admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:25.028602 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779a8e02-4190-42ed-8de0-def7f9aadffa admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:25.028872 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779a8e02-4190-42ed-8de0-def7f9aadffa admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:25.029534 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779a8e02-4190-42ed-8de0-def7f9aadffa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:25.031118 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779a8e02-4190-42ed-8de0-def7f9aadffa admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:25.041771 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1463/2911] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:31:25 2022] GET /identity/v3/roles => generated 1427 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:31:25.046050 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161b23df-3bf8-4642-b862-e30e9948b0b2 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:25.057179 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161b23df-3bf8-4642-b862-e30e9948b0b2 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:25.061801 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2a4597-e902-4355-984f-ea2b09982702 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:25.062400 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2a4597-e902-4355-984f-ea2b09982702 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:25.062688 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2a4597-e902-4355-984f-ea2b09982702 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:25.062962 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2a4597-e902-4355-984f-ea2b09982702 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:25.063607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2a4597-e902-4355-984f-ea2b09982702 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:25.065245 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2a4597-e902-4355-984f-ea2b09982702 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:25.076746 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1449/2912] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:31:25 2022] GET /identity/v3/roles => generated 1427 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:31:25.080823 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779a8e02-4190-42ed-8de0-def7f9aadffa admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:25.092239 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779a8e02-4190-42ed-8de0-def7f9aadffa admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:25.098256 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6f23e0-ec6d-4bce-a5d4-21a708ed570e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:25.099186 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6f23e0-ec6d-4bce-a5d4-21a708ed570e admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:25.099484 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6f23e0-ec6d-4bce-a5d4-21a708ed570e admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:25.099811 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6f23e0-ec6d-4bce-a5d4-21a708ed570e admin admin] PATH_INFO: `/v3/projects/cbe29d9088c0480b86dc4bd15545f60e/users/d40ab45def1c49fca5615d6768fb9b26/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:25.114751 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6f23e0-ec6d-4bce-a5d4-21a708ed570e admin admin] RBAC: Authorizing `identity:create_grant(project_id=cbe29d9088c0480b86dc4bd15545f60e, user_id=d40ab45def1c49fca5615d6768fb9b26, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:25.117512 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6f23e0-ec6d-4bce-a5d4-21a708ed570e admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:25.131480 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1464/2913] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:31:25 2022] PUT /identity/v3/projects/cbe29d9088c0480b86dc4bd15545f60e/users/d40ab45def1c49fca5615d6768fb9b26/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:31:25.141674 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e52985-aae1-4494-a8a4-9a8a07c1434a None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:25.141999 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e52985-aae1-4494-a8a4-9a8a07c1434a None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:25.142281 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e52985-aae1-4494-a8a4-9a8a07c1434a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:25.171441 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-72e52985-aae1-4494-a8a4-9a8a07c1434a None None] MFA Rules not processed for user `d40ab45def1c49fca5615d6768fb9b26`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:31:25.172685 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-72e52985-aae1-4494-a8a4-9a8a07c1434a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:25.237893 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1450/2914] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:31:25 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:31:25.242421 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6f23e0-ec6d-4bce-a5d4-21a708ed570e admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:25.305342 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6f23e0-ec6d-4bce-a5d4-21a708ed570e admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:25.310226 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba70afc-3175-4827-8371-9f58ccc8672c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:25.310833 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba70afc-3175-4827-8371-9f58ccc8672c admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:25.311117 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba70afc-3175-4827-8371-9f58ccc8672c admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:25.311396 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba70afc-3175-4827-8371-9f58ccc8672c admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:25.312068 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba70afc-3175-4827-8371-9f58ccc8672c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:25.315068 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba70afc-3175-4827-8371-9f58ccc8672c admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:25.343264 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1465/2915] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:31:25 2022] POST /identity/v3/projects => generated 386 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:31:25.347538 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72e52985-aae1-4494-a8a4-9a8a07c1434a None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:25.358981 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72e52985-aae1-4494-a8a4-9a8a07c1434a None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:25.415775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55471f77-f89e-4b4a-aa62-21c703bdc2cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:25.416368 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55471f77-f89e-4b4a-aa62-21c703bdc2cd admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:25.416648 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55471f77-f89e-4b4a-aa62-21c703bdc2cd admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:25.416913 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55471f77-f89e-4b4a-aa62-21c703bdc2cd admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:25.417535 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55471f77-f89e-4b4a-aa62-21c703bdc2cd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:25.419428 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55471f77-f89e-4b4a-aa62-21c703bdc2cd admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:25.439785 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1451/2916] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:31:25 2022] POST /identity/v3/users => generated 294 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:31:25.444787 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba70afc-3175-4827-8371-9f58ccc8672c admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:25.456068 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba70afc-3175-4827-8371-9f58ccc8672c admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:25.461920 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eada843-1e95-4398-82ef-fe576bf50348 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:25.462546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eada843-1e95-4398-82ef-fe576bf50348 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:25.462824 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eada843-1e95-4398-82ef-fe576bf50348 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:25.463095 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eada843-1e95-4398-82ef-fe576bf50348 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:25.463777 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eada843-1e95-4398-82ef-fe576bf50348 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:25.465356 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eada843-1e95-4398-82ef-fe576bf50348 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:25.471077 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:31:25.478121 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1466/2917] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:31:25 2022] GET /identity/v3/roles => generated 1427 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:31:25.482019 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55471f77-f89e-4b4a-aa62-21c703bdc2cd admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:25.492985 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55471f77-f89e-4b4a-aa62-21c703bdc2cd admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:25.497723 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d9615a-7eb4-460b-91d2-0174b26da973 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:25.498326 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d9615a-7eb4-460b-91d2-0174b26da973 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:25.498603 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d9615a-7eb4-460b-91d2-0174b26da973 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:25.498870 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d9615a-7eb4-460b-91d2-0174b26da973 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:25.499541 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d9615a-7eb4-460b-91d2-0174b26da973 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:25.501176 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d9615a-7eb4-460b-91d2-0174b26da973 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:25.512683 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1452/2918] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:31:25 2022] GET /identity/v3/roles => generated 1427 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 16:31:25.516915 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eada843-1e95-4398-82ef-fe576bf50348 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:25.528293 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eada843-1e95-4398-82ef-fe576bf50348 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:25.532905 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1753f1b9-abfa-4b18-947f-9590c3fb5a12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:25.533553 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1753f1b9-abfa-4b18-947f-9590c3fb5a12 admin admin] REQUEST_METHOD: `PUT` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:25.533838 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1753f1b9-abfa-4b18-947f-9590c3fb5a12 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:25.534118 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1753f1b9-abfa-4b18-947f-9590c3fb5a12 admin admin] PATH_INFO: `/v3/projects/80fd8be5b5f34176aca6e57e4620f6d3/users/5e4f95f4801e4d9899410516b8087ce2/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:25.548978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1753f1b9-abfa-4b18-947f-9590c3fb5a12 admin admin] RBAC: Authorizing `identity:create_grant(project_id=80fd8be5b5f34176aca6e57e4620f6d3, user_id=5e4f95f4801e4d9899410516b8087ce2, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:25.552226 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1753f1b9-abfa-4b18-947f-9590c3fb5a12 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:25.565927 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1467/2919] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:31:25 2022] PUT /identity/v3/projects/80fd8be5b5f34176aca6e57e4620f6d3/users/5e4f95f4801e4d9899410516b8087ce2/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:31:25.578180 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452ff27d-59dd-45cf-84cc-057821921eeb None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:25.578476 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452ff27d-59dd-45cf-84cc-057821921eeb None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:25.578756 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452ff27d-59dd-45cf-84cc-057821921eeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:25.608295 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-452ff27d-59dd-45cf-84cc-057821921eeb None None] MFA Rules not processed for user `5e4f95f4801e4d9899410516b8087ce2`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:31:25.609537 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-452ff27d-59dd-45cf-84cc-057821921eeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:25.651692 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:31:25.676599 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1453/2920] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:31:25 2022] POST /identity/v3/auth/tokens => generated 3272 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:31:25.681199 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1753f1b9-abfa-4b18-947f-9590c3fb5a12 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:25.744228 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1753f1b9-abfa-4b18-947f-9590c3fb5a12 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:25.748819 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab09a0c2-4692-4775-869d-920ca6fe7d20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:25.749415 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab09a0c2-4692-4775-869d-920ca6fe7d20 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:25.749695 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab09a0c2-4692-4775-869d-920ca6fe7d20 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:25.749968 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab09a0c2-4692-4775-869d-920ca6fe7d20 admin admin] PATH_INFO: `/v3/projects` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:25.750589 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab09a0c2-4692-4775-869d-920ca6fe7d20 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:25.752515 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab09a0c2-4692-4775-869d-920ca6fe7d20 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:25.779650 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1468/2921] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:31:25 2022] POST /identity/v3/projects => generated 386 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:31:25.783864 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452ff27d-59dd-45cf-84cc-057821921eeb None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:25.794992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452ff27d-59dd-45cf-84cc-057821921eeb None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:25.854062 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b281ac5f-e42e-423b-a755-189550eaa4af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:25.854681 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b281ac5f-e42e-423b-a755-189550eaa4af admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:25.854986 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b281ac5f-e42e-423b-a755-189550eaa4af admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:25.855254 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b281ac5f-e42e-423b-a755-189550eaa4af admin admin] PATH_INFO: `/v3/users` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:25.855948 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b281ac5f-e42e-423b-a755-189550eaa4af admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:25.857849 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b281ac5f-e42e-423b-a755-189550eaa4af admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:25.878787 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1454/2922] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:31:25 2022] POST /identity/v3/users => generated 300 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:31:25.883868 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab09a0c2-4692-4775-869d-920ca6fe7d20 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:25.894585 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab09a0c2-4692-4775-869d-920ca6fe7d20 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:25.899799 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09048a4f-7d39-4005-bcf8-92863cec59b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:25.900443 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09048a4f-7d39-4005-bcf8-92863cec59b2 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:25.900753 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09048a4f-7d39-4005-bcf8-92863cec59b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:25.901059 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09048a4f-7d39-4005-bcf8-92863cec59b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:25.901713 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09048a4f-7d39-4005-bcf8-92863cec59b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:25.903399 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09048a4f-7d39-4005-bcf8-92863cec59b2 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:25.914435 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1469/2923] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:31:25 2022] GET /identity/v3/roles => generated 1427 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:31:25.918669 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b281ac5f-e42e-423b-a755-189550eaa4af admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:25.929386 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b281ac5f-e42e-423b-a755-189550eaa4af admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:25.933906 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e4e48f-6c9d-4a7d-9367-7f74e8e44908 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:25.934550 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e4e48f-6c9d-4a7d-9367-7f74e8e44908 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:25.934822 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e4e48f-6c9d-4a7d-9367-7f74e8e44908 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:25.935091 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e4e48f-6c9d-4a7d-9367-7f74e8e44908 admin admin] PATH_INFO: `/v3/projects/979ba572090244ec9c0cfffee5ff39dd/users/0259c70d4df44bbaba4ed6b28947312c/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:25.950505 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e4e48f-6c9d-4a7d-9367-7f74e8e44908 admin admin] RBAC: Authorizing `identity:create_grant(project_id=979ba572090244ec9c0cfffee5ff39dd, user_id=0259c70d4df44bbaba4ed6b28947312c, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:25.953461 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e4e48f-6c9d-4a7d-9367-7f74e8e44908 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:25.966085 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1455/2924] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:31:25 2022] PUT /identity/v3/projects/979ba572090244ec9c0cfffee5ff39dd/users/0259c70d4df44bbaba4ed6b28947312c/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:31:25.970610 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09048a4f-7d39-4005-bcf8-92863cec59b2 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:25.980639 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09048a4f-7d39-4005-bcf8-92863cec59b2 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:26.035643 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab5a7e9-1d92-42f4-8690-4981613bc140 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:26.036249 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab5a7e9-1d92-42f4-8690-4981613bc140 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:26.036531 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab5a7e9-1d92-42f4-8690-4981613bc140 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:26.036804 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab5a7e9-1d92-42f4-8690-4981613bc140 admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:26.037576 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab5a7e9-1d92-42f4-8690-4981613bc140 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:26.039054 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab5a7e9-1d92-42f4-8690-4981613bc140 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:26.049297 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1470/2925] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:31:25 2022] GET /identity/v3/roles => generated 1427 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:31:26.054469 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e4e48f-6c9d-4a7d-9367-7f74e8e44908 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:26.065278 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e4e48f-6c9d-4a7d-9367-7f74e8e44908 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:26.071066 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ddad36-d991-4387-be15-1b9e5e918b2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:26.071751 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ddad36-d991-4387-be15-1b9e5e918b2d admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:26.072034 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ddad36-d991-4387-be15-1b9e5e918b2d admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:26.072307 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ddad36-d991-4387-be15-1b9e5e918b2d admin admin] PATH_INFO: `/v3/projects/979ba572090244ec9c0cfffee5ff39dd/users/0259c70d4df44bbaba4ed6b28947312c/roles/4d8fbbaea81a4145b89af12350bb067c` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:26.074214 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ddad36-d991-4387-be15-1b9e5e918b2d admin admin] RBAC: Authorizing `identity:create_grant(project_id=979ba572090244ec9c0cfffee5ff39dd, user_id=0259c70d4df44bbaba4ed6b28947312c, role_id=4d8fbbaea81a4145b89af12350bb067c)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:26.076903 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ddad36-d991-4387-be15-1b9e5e918b2d admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:26.087655 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1456/2926] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:31:26 2022] PUT /identity/v3/projects/979ba572090244ec9c0cfffee5ff39dd/users/0259c70d4df44bbaba4ed6b28947312c/roles/4d8fbbaea81a4145b89af12350bb067c => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:31:26.098725 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe3ffcc-0db7-4598-9ac2-8f2a63208eb9 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:26.099011 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe3ffcc-0db7-4598-9ac2-8f2a63208eb9 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:26.099289 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe3ffcc-0db7-4598-9ac2-8f2a63208eb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:26.126998 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-0fe3ffcc-0db7-4598-9ac2-8f2a63208eb9 None None] MFA Rules not processed for user `0259c70d4df44bbaba4ed6b28947312c`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:31:26.128234 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-0fe3ffcc-0db7-4598-9ac2-8f2a63208eb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:26.163542 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:31:26.200129 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1471/2927] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:31:26 2022] POST /identity/v3/auth/tokens => generated 3339 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:31:26.211113 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ddad36-d991-4387-be15-1b9e5e918b2d admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:26.212934 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-79ddad36-d991-4387-be15-1b9e5e918b2d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:26.272393 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ddad36-d991-4387-be15-1b9e5e918b2d admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:26.276062 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91175614-8a67-4631-9354-3c50fce3ee3a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:26.276659 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91175614-8a67-4631-9354-3c50fce3ee3a service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:26.276938 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91175614-8a67-4631-9354-3c50fce3ee3a service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:26.277212 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91175614-8a67-4631-9354-3c50fce3ee3a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:26.326823 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:31:26.338076 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91175614-8a67-4631-9354-3c50fce3ee3a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:26.339534 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91175614-8a67-4631-9354-3c50fce3ee3a service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:26.349293 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1457/2928] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:31:26 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:31:27.867357 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe3ffcc-0db7-4598-9ac2-8f2a63208eb9 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:27.868815 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-0fe3ffcc-0db7-4598-9ac2-8f2a63208eb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:27.881538 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe3ffcc-0db7-4598-9ac2-8f2a63208eb9 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:27.885119 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f10d667-7bab-4997-a9da-8efe8cf6e920 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:27.885741 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f10d667-7bab-4997-a9da-8efe8cf6e920 service cinder] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:27.886022 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f10d667-7bab-4997-a9da-8efe8cf6e920 service cinder] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:27.886296 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f10d667-7bab-4997-a9da-8efe8cf6e920 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:27.951455 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f10d667-7bab-4997-a9da-8efe8cf6e920 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:27.952841 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f10d667-7bab-4997-a9da-8efe8cf6e920 service cinder] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:27.961693 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1472/2929] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:31:27 2022] GET /identity/v3/auth/tokens => generated 3272 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:31:28.506826 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91175614-8a67-4631-9354-3c50fce3ee3a service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:28.515547 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91175614-8a67-4631-9354-3c50fce3ee3a service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:28.519095 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52a69db-2fa8-40df-aa62-63b84dd42c50 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:28.519775 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52a69db-2fa8-40df-aa62-63b84dd42c50 service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:28.520054 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52a69db-2fa8-40df-aa62-63b84dd42c50 service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:28.520329 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52a69db-2fa8-40df-aa62-63b84dd42c50 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:28.529122 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52a69db-2fa8-40df-aa62-63b84dd42c50 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:28.530544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52a69db-2fa8-40df-aa62-63b84dd42c50 service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:28.540489 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1458/2930] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:31:28 2022] GET /identity/v3/auth/tokens => generated 3339 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 16:31:32.425449 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f10d667-7bab-4997-a9da-8efe8cf6e920 service cinder] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:32.488777 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f10d667-7bab-4997-a9da-8efe8cf6e920 service cinder] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:32.493927 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4d075d-1255-43a5-a561-d589efc9cf2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:32.494564 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4d075d-1255-43a5-a561-d589efc9cf2b admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:32.494840 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4d075d-1255-43a5-a561-d589efc9cf2b admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:32.495109 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4d075d-1255-43a5-a561-d589efc9cf2b admin admin] PATH_INFO: `/v3/users/d40ab45def1c49fca5615d6768fb9b26` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:32.496235 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4d075d-1255-43a5-a561-d589efc9cf2b admin admin] RBAC: Authorizing `identity:delete_user(user_id=d40ab45def1c49fca5615d6768fb9b26)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:32.498168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4d075d-1255-43a5-a561-d589efc9cf2b admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:32.523944 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52a69db-2fa8-40df-aa62-63b84dd42c50 service cinder] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:32.524924 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-e52a69db-2fa8-40df-aa62-63b84dd42c50 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:32.549581 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-ec4d075d-1255-43a5-a561-d589efc9cf2b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd40ab45def1c49fca5615d6768fb9b26'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:32.550229 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-ec4d075d-1255-43a5-a561-d589efc9cf2b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd40ab45def1c49fca5615d6768fb9b26'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:32.550848 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-ec4d075d-1255-43a5-a561-d589efc9cf2b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd40ab45def1c49fca5615d6768fb9b26'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:32.559949 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-ec4d075d-1255-43a5-a561-d589efc9cf2b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd40ab45def1c49fca5615d6768fb9b26'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:32.573567 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-ec4d075d-1255-43a5-a561-d589efc9cf2b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd40ab45def1c49fca5615d6768fb9b26'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:32.574755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:31:32.592138 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1473/2931] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:31:32 2022] DELETE /identity/v3/users/d40ab45def1c49fca5615d6768fb9b26 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:31:32.597428 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1276c95-b4ff-4387-b5d4-bd9598647334 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:32.597967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1276c95-b4ff-4387-b5d4-bd9598647334 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:32.598431 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1276c95-b4ff-4387-b5d4-bd9598647334 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:32.615233 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52a69db-2fa8-40df-aa62-63b84dd42c50 service cinder] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:32.622324 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-a1276c95-b4ff-4387-b5d4-bd9598647334 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:31:32.623460 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-a1276c95-b4ff-4387-b5d4-bd9598647334 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:32.685175 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3495f94-94bd-432b-b48e-b3f8f989eba8 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0ee0002a76c3471f819b37fef510ef99', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:32.686140 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3495f94-94bd-432b-b48e-b3f8f989eba8 service placement] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:32.686607 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3495f94-94bd-432b-b48e-b3f8f989eba8 service placement] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:32.687059 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3495f94-94bd-432b-b48e-b3f8f989eba8 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:32.689712 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-a3495f94-94bd-432b-b48e-b3f8f989eba8 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:32.690978 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1474/2932] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:31:32 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:31:32.701288 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1276c95-b4ff-4387-b5d4-bd9598647334 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:32.702665 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-a1276c95-b4ff-4387-b5d4-bd9598647334 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:32.702978 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3495f94-94bd-432b-b48e-b3f8f989eba8 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:32.704640 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3495f94-94bd-432b-b48e-b3f8f989eba8 service placement] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:32.718015 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1459/2933] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:31:32 2022] GET /identity/v3/auth/tokens => generated 3318 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:31:32.767364 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1276c95-b4ff-4387-b5d4-bd9598647334 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:32.772201 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf09fbba-59c8-4c6a-b74e-c61667e8f749 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:32.772814 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf09fbba-59c8-4c6a-b74e-c61667e8f749 service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:32.773201 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf09fbba-59c8-4c6a-b74e-c61667e8f749 service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:32.773528 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf09fbba-59c8-4c6a-b74e-c61667e8f749 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:32.783496 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf09fbba-59c8-4c6a-b74e-c61667e8f749 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:32.784956 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf09fbba-59c8-4c6a-b74e-c61667e8f749 service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:32.795464 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1475/2934] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:31:32 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 16:31:33.071507 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3495f94-94bd-432b-b48e-b3f8f989eba8 service placement] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:33.072962 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-a3495f94-94bd-432b-b48e-b3f8f989eba8 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:33.092756 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3495f94-94bd-432b-b48e-b3f8f989eba8 service placement] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:33.099815 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c091fafc-538f-4be0-a421-aa104b3da5ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:33.100536 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c091fafc-538f-4be0-a421-aa104b3da5ca admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:33.100825 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c091fafc-538f-4be0-a421-aa104b3da5ca admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:33.101332 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c091fafc-538f-4be0-a421-aa104b3da5ca admin admin] PATH_INFO: `/v3/projects/cbe29d9088c0480b86dc4bd15545f60e` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:33.103313 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c091fafc-538f-4be0-a421-aa104b3da5ca admin admin] RBAC: Authorizing `identity:delete_project(project_id=cbe29d9088c0480b86dc4bd15545f60e)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:33.104573 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c091fafc-538f-4be0-a421-aa104b3da5ca admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:33.152848 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-c091fafc-538f-4be0-a421-aa104b3da5ca admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cbe29d9088c0480b86dc4bd15545f60e', 'request_id': 'req-c091fafc-538f-4be0-a421-aa104b3da5ca'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:33.159585 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-c091fafc-538f-4be0-a421-aa104b3da5ca admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cbe29d9088c0480b86dc4bd15545f60e', 'request_id': 'req-c091fafc-538f-4be0-a421-aa104b3da5ca'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:33.165246 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-c091fafc-538f-4be0-a421-aa104b3da5ca admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cbe29d9088c0480b86dc4bd15545f60e', 'request_id': 'req-c091fafc-538f-4be0-a421-aa104b3da5ca'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:33.167040 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:31:33.184780 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-c091fafc-538f-4be0-a421-aa104b3da5ca admin admin] The token cache is being invalidate because project cbe29d9088c0480b86dc4bd15545f60e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:31:33.185078 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-c091fafc-538f-4be0-a421-aa104b3da5ca admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:33.185790 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-c091fafc-538f-4be0-a421-aa104b3da5ca admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:33.187465 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1460/2935] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:31:33 2022] DELETE /identity/v3/projects/cbe29d9088c0480b86dc4bd15545f60e => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:31:33.191744 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf09fbba-59c8-4c6a-b74e-c61667e8f749 service neutron] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:33.204390 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf09fbba-59c8-4c6a-b74e-c61667e8f749 service neutron] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:33.260921 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc89e73-1cf2-4cca-8546-1964904c6392 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:33.261546 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc89e73-1cf2-4cca-8546-1964904c6392 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:33.261827 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc89e73-1cf2-4cca-8546-1964904c6392 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:33.262126 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc89e73-1cf2-4cca-8546-1964904c6392 admin admin] PATH_INFO: `/v3/users/5e4f95f4801e4d9899410516b8087ce2` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:33.263347 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc89e73-1cf2-4cca-8546-1964904c6392 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5e4f95f4801e4d9899410516b8087ce2)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:33.265300 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc89e73-1cf2-4cca-8546-1964904c6392 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:33.303950 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:31:33.319944 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-1dc89e73-1cf2-4cca-8546-1964904c6392 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5e4f95f4801e4d9899410516b8087ce2'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:33.319944 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-1dc89e73-1cf2-4cca-8546-1964904c6392 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5e4f95f4801e4d9899410516b8087ce2'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:33.319944 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-1dc89e73-1cf2-4cca-8546-1964904c6392 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5e4f95f4801e4d9899410516b8087ce2'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:33.327961 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-1dc89e73-1cf2-4cca-8546-1964904c6392 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5e4f95f4801e4d9899410516b8087ce2'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:33.344794 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-1dc89e73-1cf2-4cca-8546-1964904c6392 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5e4f95f4801e4d9899410516b8087ce2'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:33.359987 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1476/2936] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:31:33 2022] DELETE /identity/v3/users/5e4f95f4801e4d9899410516b8087ce2 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 16:31:33.633111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c091fafc-538f-4be0-a421-aa104b3da5ca admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:33.634780 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-c091fafc-538f-4be0-a421-aa104b3da5ca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:33.651824 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c091fafc-538f-4be0-a421-aa104b3da5ca admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:33.716789 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71193c1-becf-47ed-b66c-79437e425767 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:33.717332 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71193c1-becf-47ed-b66c-79437e425767 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:33.717610 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71193c1-becf-47ed-b66c-79437e425767 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:33.717907 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71193c1-becf-47ed-b66c-79437e425767 admin admin] PATH_INFO: `/v3/projects/80fd8be5b5f34176aca6e57e4620f6d3` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:33.719249 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e71193c1-becf-47ed-b66c-79437e425767 admin admin] RBAC: Authorizing `identity:delete_project(project_id=80fd8be5b5f34176aca6e57e4620f6d3)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:33.721211 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e71193c1-becf-47ed-b66c-79437e425767 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:33.768345 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-e71193c1-becf-47ed-b66c-79437e425767 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '80fd8be5b5f34176aca6e57e4620f6d3', 'request_id': 'req-e71193c1-becf-47ed-b66c-79437e425767'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:33.774583 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-e71193c1-becf-47ed-b66c-79437e425767 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '80fd8be5b5f34176aca6e57e4620f6d3', 'request_id': 'req-e71193c1-becf-47ed-b66c-79437e425767'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:33.780103 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-e71193c1-becf-47ed-b66c-79437e425767 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '80fd8be5b5f34176aca6e57e4620f6d3', 'request_id': 'req-e71193c1-becf-47ed-b66c-79437e425767'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:33.793050 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-e71193c1-becf-47ed-b66c-79437e425767 admin admin] The token cache is being invalidate because project 80fd8be5b5f34176aca6e57e4620f6d3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:31:33.793334 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-e71193c1-becf-47ed-b66c-79437e425767 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:33.793970 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-e71193c1-becf-47ed-b66c-79437e425767 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:33.795478 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1461/2937] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:31:33 2022] DELETE /identity/v3/projects/80fd8be5b5f34176aca6e57e4620f6d3 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:31:33.799452 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc89e73-1cf2-4cca-8546-1964904c6392 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:33.800977 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-1dc89e73-1cf2-4cca-8546-1964904c6392 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:33.815532 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc89e73-1cf2-4cca-8546-1964904c6392 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:33.874862 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0479762d-f391-4ea2-8a98-8945956bd69f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:33.875503 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0479762d-f391-4ea2-8a98-8945956bd69f admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:33.875822 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0479762d-f391-4ea2-8a98-8945956bd69f admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:33.876100 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0479762d-f391-4ea2-8a98-8945956bd69f admin admin] PATH_INFO: `/v3/users/0259c70d4df44bbaba4ed6b28947312c` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:33.877230 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0479762d-f391-4ea2-8a98-8945956bd69f admin admin] RBAC: Authorizing `identity:delete_user(user_id=0259c70d4df44bbaba4ed6b28947312c)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:33.879164 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0479762d-f391-4ea2-8a98-8945956bd69f admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:33.931278 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0479762d-f391-4ea2-8a98-8945956bd69f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0259c70d4df44bbaba4ed6b28947312c'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:33.931967 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0479762d-f391-4ea2-8a98-8945956bd69f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0259c70d4df44bbaba4ed6b28947312c'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:33.932576 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0479762d-f391-4ea2-8a98-8945956bd69f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0259c70d4df44bbaba4ed6b28947312c'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:33.940104 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0479762d-f391-4ea2-8a98-8945956bd69f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0259c70d4df44bbaba4ed6b28947312c'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:33.954650 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-0479762d-f391-4ea2-8a98-8945956bd69f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0259c70d4df44bbaba4ed6b28947312c'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:33.968895 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1477/2938] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:31:33 2022] DELETE /identity/v3/users/0259c70d4df44bbaba4ed6b28947312c => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:31:34.244663 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71193c1-becf-47ed-b66c-79437e425767 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:34.246136 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-e71193c1-becf-47ed-b66c-79437e425767 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:34.259579 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71193c1-becf-47ed-b66c-79437e425767 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:34.319002 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5baee230-786c-4c3c-97cc-8da39e84f57a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:34.319651 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5baee230-786c-4c3c-97cc-8da39e84f57a admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:34.319949 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5baee230-786c-4c3c-97cc-8da39e84f57a admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:34.320220 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5baee230-786c-4c3c-97cc-8da39e84f57a admin admin] PATH_INFO: `/v3/projects/979ba572090244ec9c0cfffee5ff39dd` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:34.321354 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5baee230-786c-4c3c-97cc-8da39e84f57a admin admin] RBAC: Authorizing `identity:delete_project(project_id=979ba572090244ec9c0cfffee5ff39dd)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:34.323252 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5baee230-786c-4c3c-97cc-8da39e84f57a admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:34.348487 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:INSERT=2,keystone:DELETE=9,keystone:SELECT=89 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:31:34.371014 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-5baee230-786c-4c3c-97cc-8da39e84f57a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '979ba572090244ec9c0cfffee5ff39dd', 'request_id': 'req-5baee230-786c-4c3c-97cc-8da39e84f57a'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:34.376983 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-5baee230-786c-4c3c-97cc-8da39e84f57a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '979ba572090244ec9c0cfffee5ff39dd', 'request_id': 'req-5baee230-786c-4c3c-97cc-8da39e84f57a'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:34.383111 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-5baee230-786c-4c3c-97cc-8da39e84f57a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '979ba572090244ec9c0cfffee5ff39dd', 'request_id': 'req-5baee230-786c-4c3c-97cc-8da39e84f57a'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:34.395652 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-5baee230-786c-4c3c-97cc-8da39e84f57a admin admin] The token cache is being invalidate because project 979ba572090244ec9c0cfffee5ff39dd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:31:34.395965 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-5baee230-786c-4c3c-97cc-8da39e84f57a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:34.397418 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-5baee230-786c-4c3c-97cc-8da39e84f57a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:34.398888 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1462/2939] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:31:34 2022] DELETE /identity/v3/projects/979ba572090244ec9c0cfffee5ff39dd => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:31:34.403805 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79cc06fb-c22e-47ef-b278-d29e6175e61b None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:34.404211 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79cc06fb-c22e-47ef-b278-d29e6175e61b None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:34.404388 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79cc06fb-c22e-47ef-b278-d29e6175e61b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:34.423810 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-79cc06fb-c22e-47ef-b278-d29e6175e61b None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:31:34.425015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-79cc06fb-c22e-47ef-b278-d29e6175e61b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:34.451789 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:31:34.490232 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1478/2940] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:31:34 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:31:34.499698 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3dec18-7766-4579-bf91-cf3f8864372a None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:34.499990 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3dec18-7766-4579-bf91-cf3f8864372a None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:34.500299 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3dec18-7766-4579-bf91-cf3f8864372a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:34.520851 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-3b3dec18-7766-4579-bf91-cf3f8864372a None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:31:34.523269 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-3b3dec18-7766-4579-bf91-cf3f8864372a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:34.531273 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1463/2941] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:31:34 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:31:34.535448 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79cc06fb-c22e-47ef-b278-d29e6175e61b None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:34.545168 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79cc06fb-c22e-47ef-b278-d29e6175e61b None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:34.549662 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80fa70c-84a5-4f97-b20c-cf07d6639ba6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:34.550245 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80fa70c-84a5-4f97-b20c-cf07d6639ba6 admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:34.550522 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80fa70c-84a5-4f97-b20c-cf07d6639ba6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:34.550791 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80fa70c-84a5-4f97-b20c-cf07d6639ba6 admin admin] PATH_INFO: `/v3/domains` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:34.551429 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80fa70c-84a5-4f97-b20c-cf07d6639ba6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88217) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 16:31:34.551767 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80fa70c-84a5-4f97-b20c-cf07d6639ba6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:34.553353 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80fa70c-84a5-4f97-b20c-cf07d6639ba6 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:34.565219 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1479/2942] 10.222.0.102 () {58 vars in 1158 bytes} [Wed Aug 24 16:31:34 2022] GET /identity/v3/domains?name=Default => generated 310 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 16:31:34.570396 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3dec18-7766-4579-bf91-cf3f8864372a None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:34.579670 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3dec18-7766-4579-bf91-cf3f8864372a None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:34.585072 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0399f000-8299-47f1-aff1-953e8ee1b577 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:34.585660 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0399f000-8299-47f1-aff1-953e8ee1b577 admin admin] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:34.585937 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0399f000-8299-47f1-aff1-953e8ee1b577 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:34.586208 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0399f000-8299-47f1-aff1-953e8ee1b577 admin admin] PATH_INFO: `/v3/projects` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:34.586832 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0399f000-8299-47f1-aff1-953e8ee1b577 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:34.588772 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0399f000-8299-47f1-aff1-953e8ee1b577 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:34.616338 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1464/2943] 10.222.0.102 () {60 vars in 1158 bytes} [Wed Aug 24 16:31:34 2022] POST /identity/v3/projects => generated 392 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 16:31:34.620544 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80fa70c-84a5-4f97-b20c-cf07d6639ba6 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:34.630441 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80fa70c-84a5-4f97-b20c-cf07d6639ba6 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:34.691935 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ebd4f7-0e56-41ac-a52a-1221316b6ef0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:34.692544 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ebd4f7-0e56-41ac-a52a-1221316b6ef0 admin admin] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:34.692832 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ebd4f7-0e56-41ac-a52a-1221316b6ef0 admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:34.693101 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ebd4f7-0e56-41ac-a52a-1221316b6ef0 admin admin] PATH_INFO: `/v3/users` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:34.694533 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ebd4f7-0e56-41ac-a52a-1221316b6ef0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:34.696522 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ebd4f7-0e56-41ac-a52a-1221316b6ef0 admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:34.719483 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1480/2944] 10.222.0.102 () {60 vars in 1149 bytes} [Wed Aug 24 16:31:34 2022] POST /identity/v3/users => generated 297 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 16:31:34.723850 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0399f000-8299-47f1-aff1-953e8ee1b577 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:34.735023 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0399f000-8299-47f1-aff1-953e8ee1b577 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:34.739712 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845301af-e2a0-40cd-ba4f-1bce0943eb81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:34.741704 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845301af-e2a0-40cd-ba4f-1bce0943eb81 admin admin] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:34.741923 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845301af-e2a0-40cd-ba4f-1bce0943eb81 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:34.742201 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845301af-e2a0-40cd-ba4f-1bce0943eb81 admin admin] PATH_INFO: `/v3/roles` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:34.742855 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845301af-e2a0-40cd-ba4f-1bce0943eb81 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:34.744493 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845301af-e2a0-40cd-ba4f-1bce0943eb81 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:34.754547 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1465/2945] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:31:34 2022] GET /identity/v3/roles => generated 1427 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 16:31:34.759237 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ebd4f7-0e56-41ac-a52a-1221316b6ef0 admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:34.769349 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ebd4f7-0e56-41ac-a52a-1221316b6ef0 admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:34.774031 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073fc0c5-5186-4a50-8e99-2e017b81503f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:34.774628 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073fc0c5-5186-4a50-8e99-2e017b81503f admin admin] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:34.774909 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073fc0c5-5186-4a50-8e99-2e017b81503f admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:34.775180 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073fc0c5-5186-4a50-8e99-2e017b81503f admin admin] PATH_INFO: `/v3/roles` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:34.775854 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073fc0c5-5186-4a50-8e99-2e017b81503f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:34.777462 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073fc0c5-5186-4a50-8e99-2e017b81503f admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:34.788169 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1481/2946] 10.222.0.102 () {58 vars in 1127 bytes} [Wed Aug 24 16:31:34 2022] GET /identity/v3/roles => generated 1427 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 16:31:34.792355 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845301af-e2a0-40cd-ba4f-1bce0943eb81 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:34.802922 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845301af-e2a0-40cd-ba4f-1bce0943eb81 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:34.807845 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9f2f9f-384a-4cfa-b655-66e30332cb37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:34.808486 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9f2f9f-384a-4cfa-b655-66e30332cb37 admin admin] REQUEST_METHOD: `PUT` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:34.808765 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9f2f9f-384a-4cfa-b655-66e30332cb37 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:34.809037 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9f2f9f-384a-4cfa-b655-66e30332cb37 admin admin] PATH_INFO: `/v3/projects/e032f6fc5a2948e5a1d9a02b83e74244/users/9985fa777d734214a1b4ef2dab950aaf/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:34.824625 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a9f2f9f-384a-4cfa-b655-66e30332cb37 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e032f6fc5a2948e5a1d9a02b83e74244, user_id=9985fa777d734214a1b4ef2dab950aaf, role_id=b5d2ed3e20ab47eeb741b4f43ab2eebb)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:34.827332 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a9f2f9f-384a-4cfa-b655-66e30332cb37 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:34.840834 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1466/2947] 10.222.0.102 () {60 vars in 1488 bytes} [Wed Aug 24 16:31:34 2022] PUT /identity/v3/projects/e032f6fc5a2948e5a1d9a02b83e74244/users/9985fa777d734214a1b4ef2dab950aaf/roles/b5d2ed3e20ab47eeb741b4f43ab2eebb => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:31:34.851356 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5ff877-8d53-4fcc-895d-cbdae47b8967 None None] REQUEST_METHOD: `POST` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:34.851795 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5ff877-8d53-4fcc-895d-cbdae47b8967 None None] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:34.851945 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5ff877-8d53-4fcc-895d-cbdae47b8967 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:34.879410 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.auth.core [None req-de5ff877-8d53-4fcc-895d-cbdae47b8967 None None] MFA Rules not processed for user `9985fa777d734214a1b4ef2dab950aaf`. Rule list: `[]` (Enabled: `True`). {{(pid=88217) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:31:34.881295 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-de5ff877-8d53-4fcc-895d-cbdae47b8967 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:34.944956 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1482/2948] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:31:34 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:31:34.955734 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9f2f9f-384a-4cfa-b655-66e30332cb37 admin admin] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:34.957436 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-5a9f2f9f-384a-4cfa-b655-66e30332cb37 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:35.027544 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9f2f9f-384a-4cfa-b655-66e30332cb37 admin admin] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:35.031158 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edabe218-c3c7-4e83-9afc-9e4b8f59f1d6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '08fbd28d9593476b8753f8d64ee64f5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:35.031796 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edabe218-c3c7-4e83-9afc-9e4b8f59f1d6 service cinder] REQUEST_METHOD: `GET` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:35.032080 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edabe218-c3c7-4e83-9afc-9e4b8f59f1d6 service cinder] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:35.032356 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edabe218-c3c7-4e83-9afc-9e4b8f59f1d6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:35.041751 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edabe218-c3c7-4e83-9afc-9e4b8f59f1d6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:35.043175 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edabe218-c3c7-4e83-9afc-9e4b8f59f1d6 service cinder] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:35.052084 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1467/2949] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:31:34 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:31:44.941440 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=85,keystone:INSERT=3 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:31:45.048045 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=68,keystone:DELETE=6,keystone:INSERT=3 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:31:54.557830 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5ff877-8d53-4fcc-895d-cbdae47b8967 None None] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:54.618994 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5ff877-8d53-4fcc-895d-cbdae47b8967 None None] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:54.623446 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9c8bf2-1c88-493e-bdb1-f9ced8f83d8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:54.624097 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9c8bf2-1c88-493e-bdb1-f9ced8f83d8a admin admin] REQUEST_METHOD: `DELETE` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:54.624375 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9c8bf2-1c88-493e-bdb1-f9ced8f83d8a admin admin] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:54.624652 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9c8bf2-1c88-493e-bdb1-f9ced8f83d8a admin admin] PATH_INFO: `/v3/users/9985fa777d734214a1b4ef2dab950aaf` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:54.625761 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9c8bf2-1c88-493e-bdb1-f9ced8f83d8a admin admin] RBAC: Authorizing `identity:delete_user(user_id=9985fa777d734214a1b4ef2dab950aaf)` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:54.627702 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9c8bf2-1c88-493e-bdb1-f9ced8f83d8a admin admin] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:54.673333 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4b9c8bf2-1c88-493e-bdb1-f9ced8f83d8a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9985fa777d734214a1b4ef2dab950aaf'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:54.674015 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4b9c8bf2-1c88-493e-bdb1-f9ced8f83d8a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9985fa777d734214a1b4ef2dab950aaf'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:54.674738 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4b9c8bf2-1c88-493e-bdb1-f9ced8f83d8a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9985fa777d734214a1b4ef2dab950aaf'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:54.682924 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4b9c8bf2-1c88-493e-bdb1-f9ced8f83d8a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9985fa777d734214a1b4ef2dab950aaf'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:54.695403 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.notifications [None req-4b9c8bf2-1c88-493e-bdb1-f9ced8f83d8a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9985fa777d734214a1b4ef2dab950aaf'} {{(pid=88217) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:54.707556 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1483/2950] 10.222.0.102 () {58 vars in 1229 bytes} [Wed Aug 24 16:31:54 2022] DELETE /identity/v3/users/9985fa777d734214a1b4ef2dab950aaf => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:31:54.712219 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb665344-e2bc-416a-ba43-2540816a40c2 None None] REQUEST_METHOD: `POST` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:54.712687 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb665344-e2bc-416a-ba43-2540816a40c2 None None] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:54.712796 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb665344-e2bc-416a-ba43-2540816a40c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:54.731912 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.auth.core [None req-fb665344-e2bc-416a-ba43-2540816a40c2 None None] MFA Rules not processed for user `3eb7904f17244b85baf19c438c53917f`. Rule list: `[]` (Enabled: `True`). {{(pid=88218) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 16:31:54.733119 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-fb665344-e2bc-416a-ba43-2540816a40c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:54.793954 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1468/2951] 10.222.0.102 () {58 vars in 963 bytes} [Wed Aug 24 16:31:54 2022] POST /identity/v3/auth/tokens => generated 3249 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 16:31:54.806005 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9c8bf2-1c88-493e-bdb1-f9ced8f83d8a admin admin] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:54.807424 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-4b9c8bf2-1c88-493e-bdb1-f9ced8f83d8a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:54.867792 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9c8bf2-1c88-493e-bdb1-f9ced8f83d8a admin admin] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:54.871381 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8bf282-5a2a-4a3f-8246-ffd0bfcf6f0f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48939674c1fd4cf599691d510fcd8858', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:54.872021 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8bf282-5a2a-4a3f-8246-ffd0bfcf6f0f service neutron] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:54.872303 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8bf282-5a2a-4a3f-8246-ffd0bfcf6f0f service neutron] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:54.872577 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8bf282-5a2a-4a3f-8246-ffd0bfcf6f0f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:54.881607 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8bf282-5a2a-4a3f-8246-ffd0bfcf6f0f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:54.883344 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8bf282-5a2a-4a3f-8246-ffd0bfcf6f0f service neutron] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:54.892887 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1484/2952] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:31:54 2022] GET /identity/v3/auth/tokens => generated 3249 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:31:55.174973 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb665344-e2bc-416a-ba43-2540816a40c2 None None] Authenticating user token {{(pid=88218) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:31:55.177834 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.fernet_utils [None req-fb665344-e2bc-416a-ba43-2540816a40c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88218) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:31:55.199276 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb665344-e2bc-416a-ba43-2540816a40c2 None None] Validating token access rules against request {{(pid=88218) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:31:55.205884 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ea9da2-3a14-45a3-bf26-c857c7fd7214 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3eb7904f17244b85baf19c438c53917f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '501a315437844349bdc02dafe337b855', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88218) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:31:55.206940 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ea9da2-3a14-45a3-bf26-c857c7fd7214 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:31:55.207277 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ea9da2-3a14-45a3-bf26-c857c7fd7214 admin admin] SCRIPT_NAME: `/identity` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:31:55.207726 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ea9da2-3a14-45a3-bf26-c857c7fd7214 admin admin] PATH_INFO: `/v3/projects/e032f6fc5a2948e5a1d9a02b83e74244` {{(pid=88218) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:31:55.208992 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ea9da2-3a14-45a3-bf26-c857c7fd7214 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e032f6fc5a2948e5a1d9a02b83e74244)` {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:31:55.211832 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ea9da2-3a14-45a3-bf26-c857c7fd7214 admin admin] RBAC: Authorization granted {{(pid=88218) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:31:55.258256 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d9ea9da2-3a14-45a3-bf26-c857c7fd7214 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e032f6fc5a2948e5a1d9a02b83e74244', 'request_id': 'req-d9ea9da2-3a14-45a3-bf26-c857c7fd7214'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:55.264540 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d9ea9da2-3a14-45a3-bf26-c857c7fd7214 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e032f6fc5a2948e5a1d9a02b83e74244', 'request_id': 'req-d9ea9da2-3a14-45a3-bf26-c857c7fd7214'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:55.273437 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d9ea9da2-3a14-45a3-bf26-c857c7fd7214 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e032f6fc5a2948e5a1d9a02b83e74244', 'request_id': 'req-d9ea9da2-3a14-45a3-bf26-c857c7fd7214'} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:55.287488 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d9ea9da2-3a14-45a3-bf26-c857c7fd7214 admin admin] The token cache is being invalidate because project e032f6fc5a2948e5a1d9a02b83e74244 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88218) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 16:31:55.287950 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d9ea9da2-3a14-45a3-bf26-c857c7fd7214 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:55.289053 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG keystone.notifications [None req-d9ea9da2-3a14-45a3-bf26-c857c7fd7214 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88218) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 16:31:55.291475 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: [pid: 88218|app: 0|req: 1469/2953] 10.222.0.102 () {58 vars in 1238 bytes} [Wed Aug 24 16:31:55 2022] DELETE /identity/v3/projects/e032f6fc5a2948e5a1d9a02b83e74244 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 16:32:04.888639 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=79,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:32:05.283881 dv-cinder-849325-3-iscsi devstack@keystone.service[88218]: DEBUG dbcounter [-] [88218] Writing DB stats keystone:SELECT=54,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88218) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 16:36:39.506755 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8bf282-5a2a-4a3f-8246-ffd0bfcf6f0f service neutron] Authenticating user token {{(pid=88217) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 16:36:39.508189 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-ad8bf282-5a2a-4a3f-8246-ffd0bfcf6f0f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:36:39.622360 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8bf282-5a2a-4a3f-8246-ffd0bfcf6f0f service neutron] Validating token access rules against request {{(pid=88217) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 16:36:39.684726 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b6478e-61e0-40b2-9273-51de947affb0 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0ee0002a76c3471f819b37fef510ef99', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3e50e6f6c1f44c648337c8d37a8e94a5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88217) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 16:36:39.685329 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b6478e-61e0-40b2-9273-51de947affb0 service placement] REQUEST_METHOD: `GET` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 16:36:39.685615 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b6478e-61e0-40b2-9273-51de947affb0 service placement] SCRIPT_NAME: `/identity` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 16:36:39.686022 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b6478e-61e0-40b2-9273-51de947affb0 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88217) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 16:36:39.687676 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.fernet_utils [None req-e9b6478e-61e0-40b2-9273-51de947affb0 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88217) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 16:36:39.699273 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b6478e-61e0-40b2-9273-51de947affb0 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 16:36:39.700899 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b6478e-61e0-40b2-9273-51de947affb0 service placement] RBAC: Authorization granted {{(pid=88217) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 16:36:39.711263 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: [pid: 88217|app: 0|req: 1485/2954] 10.222.0.102 () {60 vars in 1370 bytes} [Wed Aug 24 16:36:39 2022] GET /identity/v3/auth/tokens => generated 3318 bytes in 205 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 16:36:49.706479 dv-cinder-849325-3-iscsi devstack@keystone.service[88217]: DEBUG dbcounter [-] [88217] Writing DB stats keystone:SELECT=74 {{(pid=88217) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}